{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HQZu6AFaHodu"
   },
   "source": [
    "## Multisequence Alignment (MSA)\n",
    "\n",
    "Proteins are made up of sequences of amino acids chained together. Their amino acid sequence determines their structure and function. Finding proteins with similar sequences, or homologous proteins, is very useful in identifying the structures and functions of newly discovered proteins as well as identifying their ancestry. Below is an example of what a protein amino acid multisequence alignment may look like, taken from [2]."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VALKXP92I8N_"
   },
   "source": [
    "![MSA.png]()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To understand Multiple Sequence Alignment (MSA), it's helpful to first grasp pairwise sequence alignment. Pairwise sequence alignment is a hypothesis about how two sequences may have evolved from a common ancestor through events such as mutation, insertion, and deletion. When a nucleotide is aligned with a gap, it represents an indel event—either a deletion in one sequence or an insertion in the other. When two different nucleotides are aligned, this is typically interpreted as a substitution or mutation event introduced in one or both of the lineages since the time they diverged from one another.If identical nucleotides are aligned, it suggests a conserved region, which may indicate functional importance and possibly homology—evidence that the sequences share a common ancestor. Pairwise alignment also provides an optimal alignment of two sequences by strategically introducing gaps, making it useful for comparing sequences and identifying conserved regions.The alignment is an optimal hypothesis but may not reflect the actual evolutionary path."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using profile-sequence comparison instead of just sequence-sequence comparison when constructing multiple sequence alignment utilizes more information. Sequence profiles are based on frequencies of each of the 20 amino acids at each position in a sequence. Sequence profiles (or PSSMs) tell us how likely it is that a particular amino acid (or nucleotide in DNA/RNA sequences) at a specific position is due to conservation rather than random chance.This is achieved through the use of log-odds scores that compare the observed frequency of an amino acid at a particular position in the multiple sequence alignment (MSA) to its expected frequency under random conditions (i.e., the background probability)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is how the position frequence matrix looks. Sequence profile is constructed through the log odds score of these frequencies.\n",
    "<img src=\"https://web.stanford.edu/class/sbio228/public/lectures/lec9/Lecture9/Fold_Recognition/images/Use_a_Sequence_Profile.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Profile Hidden Markov Model is a probabilistic model that represents the sequence conservation at each position (including insertions and deletions) and the likelihood of transitioning between different sequence states."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Colab\n",
    "\n",
    "This tutorial and the rest in this sequence can be done in Google colab. If you'd like to open this notebook in colab, you can use the following link.\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/deepchem/deepchem/blob/master/examples/tutorials/Multisequence_Alignments.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vz_Spjk-I8rk"
   },
   "source": [
    "## HH-suite\n",
    "This tutorial will show you the basics of how to use hh-suite. [hh-suite](https://github.com/soedinglab/hh-suite) is an open source package for searching protein sequence alignments for homologous proteins. It is the current state of the art for building highly accurate multisequence alignments (MSA) from a single sequence or from MSAs.\n",
    "\n",
    "HH-suite leverages profile HMMs to improve the accuracy of detecting remote homologs (sequences that share a common ancestor but are highly diverged).\n",
    "\n",
    "Instead of comparing a single sequence against a database of sequences, it aligns one profile HMM against another profile HMM.The idea is that by comparing two profile HMMs, it can detect relationships between sequence families that are not apparent when comparing just sequences. This is particularly useful when dealing with highly diverged proteins or detecting remote homologs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "P5mrPj00f4Yv"
   },
   "source": [
    "## Setup\n",
    "\n",
    "Let's start by importing the deepchem sequence_utils module and downloading a database to compare our query sequence to.\n",
    "\n",
    "hh-suite provides a set of HMM databases that will work with the software, which you can find here: http://wwwuser.gwdg.de/~compbiol/data/hhsuite/databases/hhsuite_dbs\n",
    "\n",
    "dbCAN is a good one for this tutorial because it is a relatively smaller download. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deepchem.utils import sequence_utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dbCAN-fam-V9_a3m.ffdata\n",
      "dbCAN-fam-V9_a3m.ffindex\n",
      "dbCAN-fam-V9_hhm.ffdata\n",
      "dbCAN-fam-V9_hhm.ffindex\n",
      "dbCAN-fam-V9_cs219.ffdata\n",
      "dbCAN-fam-V9_cs219.ffindex\n",
      "dbCAN-fam-V9.md5sum\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "--2022-02-11 12:47:57--  http://wwwuser.gwdg.de/~compbiol/data/hhsuite/databases/hhsuite_dbs/dbCAN-fam-V9.tar.gz\n",
      "Resolving wwwuser.gwdg.de (wwwuser.gwdg.de)... 134.76.10.111\n",
      "Connecting to wwwuser.gwdg.de (wwwuser.gwdg.de)|134.76.10.111|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 25882327 (25M) [application/x-gzip]\n",
      "Saving to: ‘dbCAN-fam-V9.tar.gz’\n",
      "\n",
      "     0K .......... .......... .......... .......... ..........  0%  195K 2m10s\n",
      "    50K .......... .......... .......... .......... ..........  0%  391K 97s\n",
      "   100K .......... .......... .......... .......... ..........  0% 55.2M 65s\n",
      "   150K .......... .......... .......... .......... ..........  0%  392K 64s\n",
      "   200K .......... .......... .......... .......... ..........  0%  127M 51s\n",
      "   250K .......... .......... .......... .......... ..........  1% 50.8M 43s\n",
      "   300K .......... .......... .......... .......... ..........  1% 85.5M 37s\n",
      "   350K .......... .......... .......... .......... ..........  1%  395K 40s\n",
      "   400K .......... .......... .......... .......... ..........  1%  102M 35s\n",
      "   450K .......... .......... .......... .......... ..........  1% 55.2M 32s\n",
      "   500K .......... .......... .......... .......... ..........  2%  392K 35s\n",
      "   550K .......... .......... .......... .......... ..........  2% 74.8M 32s\n",
      "   600K .......... .......... .......... .......... ..........  2%  102M 29s\n",
      "   650K .......... .......... .......... .......... ..........  2%  399K 32s\n",
      "   700K .......... .......... .......... .......... ..........  2% 23.6M 29s\n",
      "   750K .......... .......... .......... .......... ..........  3% 39.5M 28s\n",
      "   800K .......... .......... .......... .......... ..........  3% 1.26M 27s\n",
      "   850K .......... .......... .......... .......... ..........  3%  571K 28s\n",
      "   900K .......... .......... .......... .......... ..........  3% 35.9M 26s\n",
      "   950K .......... .......... .......... .......... ..........  3% 52.9M 25s\n",
      "  1000K .......... .......... .......... .......... ..........  4%  401K 27s\n",
      "  1050K .......... .......... .......... .......... ..........  4% 17.2M 25s\n",
      "  1100K .......... .......... .......... .......... ..........  4% 64.7M 24s\n",
      "  1150K .......... .......... .......... .......... ..........  4%  406K 26s\n",
      "  1200K .......... .......... .......... .......... ..........  4% 14.1M 25s\n",
      "  1250K .......... .......... .......... .......... ..........  5% 45.9M 24s\n",
      "  1300K .......... .......... .......... .......... ..........  5% 1.30M 23s\n",
      "  1350K .......... .......... .......... .......... ..........  5%  572K 24s\n",
      "  1400K .......... .......... .......... .......... ..........  5% 21.6M 23s\n",
      "  1450K .......... .......... .......... .......... ..........  5% 39.8M 22s\n",
      "  1500K .......... .......... .......... .......... ..........  6%  405K 24s\n",
      "  1550K .......... .......... .......... .......... ..........  6% 16.8M 23s\n",
      "  1600K .......... .......... .......... .......... ..........  6% 28.4M 22s\n",
      "  1650K .......... .......... .......... .......... ..........  6%  413K 23s\n",
      "  1700K .......... .......... .......... .......... ..........  6% 16.0M 22s\n",
      "  1750K .......... .......... .......... .......... ..........  7% 19.0M 22s\n",
      "  1800K .......... .......... .......... .......... ..........  7% 31.5M 21s\n",
      "  1850K .......... .......... .......... .......... ..........  7%  407K 22s\n",
      "  1900K .......... .......... .......... .......... ..........  7% 14.5M 21s\n",
      "  1950K .......... .......... .......... .......... ..........  7% 40.5M 21s\n",
      "  2000K .......... .......... .......... .......... ..........  8%  410K 22s\n",
      "  2050K .......... .......... .......... .......... ..........  8% 10.1M 21s\n",
      "  2100K .......... .......... .......... .......... ..........  8% 66.8M 21s\n",
      "  2150K .......... .......... .......... .......... ..........  8% 44.7M 20s\n",
      "  2200K .......... .......... .......... .......... ..........  8%  409K 21s\n",
      "  2250K .......... .......... .......... .......... ..........  9% 11.5M 21s\n",
      "  2300K .......... .......... .......... .......... ..........  9% 64.3M 20s\n",
      "  2350K .......... .......... .......... .......... ..........  9%  389K 21s\n",
      "  2400K .......... .......... .......... .......... ..........  9%  111M 20s\n",
      "  2450K .......... .......... .......... .......... ..........  9% 63.7M 20s\n",
      "  2500K .......... .......... .......... .......... .......... 10%  110M 19s\n",
      "  2550K .......... .......... .......... .......... .......... 10%  392K 20s\n",
      "  2600K .......... .......... .......... .......... .......... 10%  114M 20s\n",
      "  2650K .......... .......... .......... .......... .......... 10% 52.9M 19s\n",
      "  2700K .......... .......... .......... .......... .......... 10%  398K 20s\n",
      "  2750K .......... .......... .......... .......... .......... 11% 31.7M 20s\n",
      "  2800K .......... .......... .......... .......... .......... 11% 79.2M 19s\n",
      "  2850K .......... .......... .......... .......... .......... 11% 71.9M 19s\n",
      "  2900K .......... .......... .......... .......... .......... 11%  399K 19s\n",
      "  2950K .......... .......... .......... .......... .......... 11% 26.2M 19s\n",
      "  3000K .......... .......... .......... .......... .......... 12% 75.7M 19s\n",
      "  3050K .......... .......... .......... .......... .......... 12% 1.46M 19s\n",
      "  3100K .......... .......... .......... .......... .......... 12%  537K 19s\n",
      "  3150K .......... .......... .......... .......... .......... 12% 34.0M 19s\n",
      "  3200K .......... .......... .......... .......... .......... 12% 78.4M 18s\n",
      "  3250K .......... .......... .......... .......... .......... 13%  402K 19s\n",
      "  3300K .......... .......... .......... .......... .......... 13% 26.9M 18s\n",
      "  3350K .......... .......... .......... .......... .......... 13% 30.2M 18s\n",
      "  3400K .......... .......... .......... .......... .......... 13% 52.6M 18s\n",
      "  3450K .......... .......... .......... .......... .......... 13%  400K 18s\n",
      "  3500K .......... .......... .......... .......... .......... 14% 41.8M 18s\n",
      "  3550K .......... .......... .......... .......... .......... 14% 46.1M 18s\n",
      "  3600K .......... .......... .......... .......... .......... 14% 1.44M 18s\n",
      "  3650K .......... .......... .......... .......... .......... 14%  544K 18s\n",
      "  3700K .......... .......... .......... .......... .......... 14% 40.0M 18s\n",
      "  3750K .......... .......... .......... .......... .......... 15% 35.8M 17s\n",
      "  3800K .......... .......... .......... .......... .......... 15%  402K 18s\n",
      "  3850K .......... .......... .......... .......... .......... 15% 46.9M 18s\n",
      "  3900K .......... .......... .......... .......... .......... 15% 19.3M 17s\n",
      "  3950K .......... .......... .......... .......... .......... 15% 82.5M 17s\n",
      "  4000K .......... .......... .......... .......... .......... 16%  404K 17s\n",
      "  4050K .......... .......... .......... .......... .......... 16% 43.3M 17s\n",
      "  4100K .......... .......... .......... .......... .......... 16% 17.1M 17s\n",
      "  4150K .......... .......... .......... .......... .......... 16% 81.5M 17s\n",
      "  4200K .......... .......... .......... .......... .......... 16%  398K 17s\n",
      "  4250K .......... .......... .......... .......... .......... 17% 50.2M 17s\n",
      "  4300K .......... .......... .......... .......... .......... 17% 45.0M 17s\n",
      "  4350K .......... .......... .......... .......... .......... 17% 1.48M 17s\n",
      "  4400K .......... .......... .......... .......... .......... 17%  539K 17s\n",
      "  4450K .......... .......... .......... .......... .......... 17% 42.5M 17s\n",
      "  4500K .......... .......... .......... .......... .......... 18% 32.2M 16s\n",
      "  4550K .......... .......... .......... .......... .......... 18% 1.51M 16s\n",
      "  4600K .......... .......... .......... .......... .......... 18%  539K 16s\n",
      "  4650K .......... .......... .......... .......... .......... 18% 36.4M 16s\n",
      "  4700K .......... .......... .......... .......... .......... 18% 34.1M 16s\n",
      "  4750K .......... .......... .......... .......... .......... 18%  440K 16s\n",
      "  4800K .......... .......... .......... .......... .......... 19% 4.12M 16s\n",
      "  4850K .......... .......... .......... .......... .......... 19% 38.7M 16s\n",
      "  4900K .......... .......... .......... .......... .......... 19% 44.3M 16s\n",
      "  4950K .......... .......... .......... .......... .......... 19%  406K 16s\n",
      "  5000K .......... .......... .......... .......... .......... 19% 21.4M 16s\n",
      "  5050K .......... .......... .......... .......... .......... 20% 39.1M 16s\n",
      "  5100K .......... .......... .......... .......... .......... 20% 43.1M 16s\n",
      "  5150K .......... .......... .......... .......... .......... 20%  407K 16s\n",
      "  5200K .......... .......... .......... .......... .......... 20% 13.1M 16s\n",
      "  5250K .......... .......... .......... .......... .......... 20% 43.6M 15s\n",
      "  5300K .......... .......... .......... .......... .......... 21% 62.2M 15s\n",
      "  5350K .......... .......... .......... .......... .......... 21%  399K 16s\n",
      "  5400K .......... .......... .......... .......... .......... 21% 58.5M 15s\n",
      "  5450K .......... .......... .......... .......... .......... 21% 38.1M 15s\n",
      "  5500K .......... .......... .......... .......... .......... 21% 55.0M 15s\n",
      "  5550K .......... .......... .......... .......... .......... 22%  401K 15s\n",
      "  5600K .......... .......... .......... .......... .......... 22% 27.5M 15s\n",
      "  5650K .......... .......... .......... .......... .......... 22% 50.0M 15s\n",
      "  5700K .......... .......... .......... .......... .......... 22% 47.0M 15s\n",
      "  5750K .......... .......... .......... .......... .......... 22%  403K 15s\n",
      "  5800K .......... .......... .......... .......... .......... 23% 21.3M 15s\n",
      "  5850K .......... .......... .......... .......... .......... 23% 29.3M 15s\n",
      "  5900K .......... .......... .......... .......... .......... 23% 59.4M 15s\n",
      "  5950K .......... .......... .......... .......... .......... 23%  406K 15s\n",
      "  6000K .......... .......... .......... .......... .......... 23% 19.4M 15s\n",
      "  6050K .......... .......... .......... .......... .......... 24% 34.9M 15s\n",
      "  6100K .......... .......... .......... .......... .......... 24% 39.6M 14s\n",
      "  6150K .......... .......... .......... .......... .......... 24%  408K 15s\n",
      "  6200K .......... .......... .......... .......... .......... 24% 18.4M 14s\n",
      "  6250K .......... .......... .......... .......... .......... 24% 36.5M 14s\n",
      "  6300K .......... .......... .......... .......... .......... 25% 29.7M 14s\n",
      "  6350K .......... .......... .......... .......... .......... 25%  420K 14s\n",
      "  6400K .......... .......... .......... .......... .......... 25% 9.03M 14s\n",
      "  6450K .......... .......... .......... .......... .......... 25% 26.0M 14s\n",
      "  6500K .......... .......... .......... .......... .......... 25% 37.5M 14s\n",
      "  6550K .......... .......... .......... .......... .......... 26%  453K 14s\n",
      "  6600K .......... .......... .......... .......... .......... 26% 3.58M 14s\n",
      "  6650K .......... .......... .......... .......... .......... 26% 18.8M 14s\n",
      "  6700K .......... .......... .......... .......... .......... 26% 40.3M 14s\n",
      "  6750K .......... .......... .......... .......... .......... 26%  455K 14s\n",
      "  6800K .......... .......... .......... .......... .......... 27% 3.64M 14s\n",
      "  6850K .......... .......... .......... .......... .......... 27% 15.0M 14s\n",
      "  6900K .......... .......... .......... .......... .......... 27% 49.0M 14s\n",
      "  6950K .......... .......... .......... .......... .......... 27%  733K 14s\n",
      "  7000K .......... .......... .......... .......... .......... 27%  896K 14s\n",
      "  7050K .......... .......... .......... .......... .......... 28% 23.0M 13s\n",
      "  7100K .......... .......... .......... .......... .......... 28% 27.6M 13s\n",
      "  7150K .......... .......... .......... .......... .......... 28% 1.59M 13s\n",
      "  7200K .......... .......... .......... .......... .......... 28%  537K 13s\n",
      "  7250K .......... .......... .......... .......... .......... 28% 24.2M 13s\n",
      "  7300K .......... .......... .......... .......... .......... 29% 23.2M 13s\n",
      "  7350K .......... .......... .......... .......... .......... 29% 68.7M 13s\n",
      "  7400K .......... .......... .......... .......... .......... 29%  408K 13s\n",
      "  7450K .......... .......... .......... .......... .......... 29% 29.0M 13s\n",
      "  7500K .......... .......... .......... .......... .......... 29% 13.4M 13s\n",
      "  7550K .......... .......... .......... .......... .......... 30% 46.7M 13s\n",
      "  7600K .......... .......... .......... .......... .......... 30%  413K 13s\n",
      "  7650K .......... .......... .......... .......... .......... 30% 19.8M 13s\n",
      "  7700K .......... .......... .......... .......... .......... 30% 29.5M 13s\n",
      "  7750K .......... .......... .......... .......... .......... 30% 17.0M 13s\n",
      "  7800K .......... .......... .......... .......... .......... 31%  463K 13s\n",
      "  7850K .......... .......... .......... .......... .......... 31% 3.53M 13s\n",
      "  7900K .......... .......... .......... .......... .......... 31% 19.9M 13s\n",
      "  7950K .......... .......... .......... .......... .......... 31% 13.7M 12s\n",
      "  8000K .......... .......... .......... .......... .......... 31% 1.72M 12s\n",
      "  8050K .......... .......... .......... .......... .......... 32%  539K 12s\n",
      "  8100K .......... .......... .......... .......... .......... 32% 13.4M 12s\n",
      "  8150K .......... .......... .......... .......... .......... 32% 21.9M 12s\n",
      "  8200K .......... .......... .......... .......... .......... 32% 43.8M 12s\n",
      "  8250K .......... .......... .......... .......... .......... 32%  415K 12s\n",
      "  8300K .......... .......... .......... .......... .......... 33% 14.1M 12s\n",
      "  8350K .......... .......... .......... .......... .......... 33% 25.5M 12s\n",
      "  8400K .......... .......... .......... .......... .......... 33% 23.7M 12s\n",
      "  8450K .......... .......... .......... .......... .......... 33%  416K 12s\n",
      "  8500K .......... .......... .......... .......... .......... 33% 45.8M 12s\n",
      "  8550K .......... .......... .......... .......... .......... 34% 13.0M 12s\n",
      "  8600K .......... .......... .......... .......... .......... 34% 19.6M 12s\n",
      "  8650K .......... .......... .......... .......... .......... 34%  765K 12s\n",
      "  8700K .......... .......... .......... .......... .......... 34%  895K 12s\n",
      "  8750K .......... .......... .......... .......... .......... 34% 14.2M 12s\n",
      "  8800K .......... .......... .......... .......... .......... 35% 18.2M 12s\n",
      "  8850K .......... .......... .......... .......... .......... 35% 43.9M 12s\n",
      "  8900K .......... .......... .......... .......... .......... 35%  416K 12s\n",
      "  8950K .......... .......... .......... .......... .......... 35% 15.8M 12s\n",
      "  9000K .......... .......... .......... .......... .......... 35% 32.0M 11s\n",
      "  9050K .......... .......... .......... .......... .......... 36% 12.4M 11s\n",
      "  9100K .......... .......... .......... .......... .......... 36%  439K 12s\n",
      "  9150K .......... .......... .......... .......... .......... 36% 7.92M 11s\n",
      "  9200K .......... .......... .......... .......... .......... 36% 14.8M 11s\n",
      "  9250K .......... .......... .......... .......... .......... 36% 11.0M 11s\n",
      "  9300K .......... .......... .......... .......... .......... 36% 1.85M 11s\n",
      "  9350K .......... .......... .......... .......... .......... 37%  537K 11s\n",
      "  9400K .......... .......... .......... .......... .......... 37% 16.8M 11s\n",
      "  9450K .......... .......... .......... .......... .......... 37% 25.4M 11s\n",
      "  9500K .......... .......... .......... .......... .......... 37% 12.7M 11s\n",
      "  9550K .......... .......... .......... .......... .......... 37%  422K 11s\n",
      "  9600K .......... .......... .......... .......... .......... 38% 38.8M 11s\n",
      "  9650K .......... .......... .......... .......... .......... 38% 14.7M 11s\n",
      "  9700K .......... .......... .......... .......... .......... 38% 11.1M 11s\n",
      "  9750K .......... .......... .......... .......... .......... 38% 1.84M 11s\n",
      "  9800K .......... .......... .......... .......... .......... 38%  538K 11s\n",
      "  9850K .......... .......... .......... .......... .......... 39% 15.3M 11s\n",
      "  9900K .......... .......... .......... .......... .......... 39% 31.6M 11s\n",
      "  9950K .......... .......... .......... .......... .......... 39% 12.4M 11s\n",
      " 10000K .......... .......... .......... .......... .......... 39%  422K 11s\n",
      " 10050K .......... .......... .......... .......... .......... 39% 14.1M 11s\n",
      " 10100K .......... .......... .......... .......... .......... 40% 30.8M 11s\n",
      " 10150K .......... .......... .......... .......... .......... 40% 12.6M 10s\n",
      " 10200K .......... .......... .......... .......... .......... 40% 1.84M 10s\n",
      " 10250K .......... .......... .......... .......... .......... 40%  532K 10s\n",
      " 10300K .......... .......... .......... .......... .......... 40% 22.0M 10s\n",
      " 10350K .......... .......... .......... .......... .......... 41% 25.4M 10s\n",
      " 10400K .......... .......... .......... .......... .......... 41% 11.9M 10s\n",
      " 10450K .......... .......... .......... .......... .......... 41%  471K 10s\n",
      " 10500K .......... .......... .......... .......... .......... 41% 3.24M 10s\n",
      " 10550K .......... .......... .......... .......... .......... 41% 17.7M 10s\n",
      " 10600K .......... .......... .......... .......... .......... 42% 13.3M 10s\n",
      " 10650K .......... .......... .......... .......... .......... 42% 1.92M 10s\n",
      " 10700K .......... .......... .......... .......... .......... 42%  532K 10s\n",
      " 10750K .......... .......... .......... .......... .......... 42% 12.6M 10s\n",
      " 10800K .......... .......... .......... .......... .......... 42% 39.9M 10s\n",
      " 10850K .......... .......... .......... .......... .......... 43% 12.8M 10s\n",
      " 10900K .......... .......... .......... .......... .......... 43%  803K 10s\n",
      " 10950K .......... .......... .......... .......... .......... 43%  849K 10s\n",
      " 11000K .......... .......... .......... .......... .......... 43% 23.2M 10s\n",
      " 11050K .......... .......... .......... .......... .......... 43% 22.8M 10s\n",
      " 11100K .......... .......... .......... .......... .......... 44% 11.8M 10s\n",
      " 11150K .......... .......... .......... .......... .......... 44%  431K 10s\n",
      " 11200K .......... .......... .......... .......... .......... 44% 12.4M 10s\n",
      " 11250K .......... .......... .......... .......... .......... 44% 17.2M 10s\n",
      " 11300K .......... .......... .......... .......... .......... 44% 14.0M 9s\n",
      " 11350K .......... .......... .......... .......... .......... 45% 1.87M 9s\n",
      " 11400K .......... .......... .......... .......... .......... 45%  533K 9s\n",
      " 11450K .......... .......... .......... .......... .......... 45% 13.6M 9s\n",
      " 11500K .......... .......... .......... .......... .......... 45% 43.6M 9s\n",
      " 11550K .......... .......... .......... .......... .......... 45% 14.5M 9s\n",
      " 11600K .......... .......... .......... .......... .......... 46%  819K 9s\n",
      " 11650K .......... .......... .......... .......... .......... 46%  839K 9s\n",
      " 11700K .......... .......... .......... .......... .......... 46% 16.7M 9s\n",
      " 11750K .......... .......... .......... .......... .......... 46% 40.2M 9s\n",
      " 11800K .......... .......... .......... .......... .......... 46% 11.6M 9s\n",
      " 11850K .......... .......... .......... .......... .......... 47%  453K 9s\n",
      " 11900K .......... .......... .......... .......... .......... 47% 4.96M 9s\n",
      " 11950K .......... .......... .......... .......... .......... 47% 18.2M 9s\n",
      " 12000K .......... .......... .......... .......... .......... 47% 17.6M 9s\n",
      " 12050K .......... .......... .......... .......... .......... 47% 1.87M 9s\n",
      " 12100K .......... .......... .......... .......... .......... 48%  533K 9s\n",
      " 12150K .......... .......... .......... .......... .......... 48% 11.7M 9s\n",
      " 12200K .......... .......... .......... .......... .......... 48% 35.4M 9s\n",
      " 12250K .......... .......... .......... .......... .......... 48% 15.3M 9s\n",
      " 12300K .......... .......... .......... .......... .......... 48% 1.88M 9s\n",
      " 12350K .......... .......... .......... .......... .......... 49%  533K 9s\n",
      " 12400K .......... .......... .......... .......... .......... 49% 12.8M 9s\n",
      " 12450K .......... .......... .......... .......... .......... 49% 22.3M 9s\n",
      " 12500K .......... .......... .......... .......... .......... 49% 21.2M 8s\n",
      " 12550K .......... .......... .......... .......... .......... 49%  806K 8s\n",
      " 12600K .......... .......... .......... .......... .......... 50%  859K 8s\n",
      " 12650K .......... .......... .......... .......... .......... 50% 15.8M 8s\n",
      " 12700K .......... .......... .......... .......... .......... 50% 14.4M 8s\n",
      " 12750K .......... .......... .......... .......... .......... 50% 21.4M 8s\n",
      " 12800K .......... .......... .......... .......... .......... 50%  484K 8s\n",
      " 12850K .......... .......... .......... .......... .......... 51% 3.11M 8s\n",
      " 12900K .......... .......... .......... .......... .......... 51% 12.4M 8s\n",
      " 12950K .......... .......... .......... .......... .......... 51% 18.9M 8s\n",
      " 13000K .......... .......... .......... .......... .......... 51% 20.2M 8s\n",
      " 13050K .......... .......... .......... .......... .......... 51%  429K 8s\n",
      " 13100K .......... .......... .......... .......... .......... 52% 11.1M 8s\n",
      " 13150K .......... .......... .......... .......... .......... 52% 26.1M 8s\n",
      " 13200K .......... .......... .......... .......... .......... 52% 17.4M 8s\n",
      " 13250K .......... .......... .......... .......... .......... 52% 1.98M 8s\n",
      " 13300K .......... .......... .......... .......... .......... 52%  532K 8s\n",
      " 13350K .......... .......... .......... .......... .......... 53% 8.56M 8s\n",
      " 13400K .......... .......... .......... .......... .......... 53% 18.6M 8s\n",
      " 13450K .......... .......... .......... .......... .......... 53% 37.9M 8s\n",
      " 13500K .......... .......... .......... .......... .......... 53% 1.83M 8s\n",
      " 13550K .......... .......... .......... .......... .......... 53%  518K 8s\n",
      " 13600K .......... .......... .......... .......... .......... 54% 53.3M 8s\n",
      " 13650K .......... .......... .......... .......... .......... 54% 17.0M 8s\n",
      " 13700K .......... .......... .......... .......... .......... 54% 32.4M 8s\n",
      " 13750K .......... .......... .......... .......... .......... 54% 1.88M 7s\n",
      " 13800K .......... .......... .......... .......... .......... 54%  518K 8s\n",
      " 13850K .......... .......... .......... .......... .......... 54% 35.2M 7s\n",
      " 13900K .......... .......... .......... .......... .......... 55% 20.6M 7s\n",
      " 13950K .......... .......... .......... .......... .......... 55% 26.7M 7s\n",
      " 14000K .......... .......... .......... .......... .......... 55% 1.90M 7s\n",
      " 14050K .......... .......... .......... .......... .......... 55%  508K 7s\n",
      " 14100K .......... .......... .......... .......... .......... 55%  111M 7s\n",
      " 14150K .......... .......... .......... .......... .......... 56% 21.6M 7s\n",
      " 14200K .......... .......... .......... .......... .......... 56% 43.4M 7s\n",
      " 14250K .......... .......... .......... .......... .......... 56% 1.90M 7s\n",
      " 14300K .......... .......... .......... .......... .......... 56%  509K 7s\n",
      " 14350K .......... .......... .......... .......... .......... 56% 47.6M 7s\n",
      " 14400K .......... .......... .......... .......... .......... 57% 29.2M 7s\n",
      " 14450K .......... .......... .......... .......... .......... 57% 36.3M 7s\n",
      " 14500K .......... .......... .......... .......... .......... 57% 1.07M 7s\n",
      " 14550K .......... .......... .......... .......... .......... 57%  639K 7s\n",
      " 14600K .......... .......... .......... .......... .......... 57% 19.1M 7s\n",
      " 14650K .......... .......... .......... .......... .......... 58%  113M 7s\n",
      " 14700K .......... .......... .......... .......... .......... 58% 40.6M 7s\n",
      " 14750K .......... .......... .......... .......... .......... 58% 1.08M 7s\n",
      " 14800K .......... .......... .......... .......... .......... 58%  638K 7s\n",
      " 14850K .......... .......... .......... .......... .......... 58% 16.9M 7s\n",
      " 14900K .......... .......... .......... .......... .......... 59% 50.2M 7s\n",
      " 14950K .......... .......... .......... .......... .......... 59% 16.4M 7s\n",
      " 15000K .......... .......... .......... .......... .......... 59% 2.03M 7s\n",
      " 15050K .......... .......... .......... .......... .......... 59%  514K 7s\n",
      " 15100K .......... .......... .......... .......... .......... 59% 18.5M 7s\n",
      " 15150K .......... .......... .......... .......... .......... 60% 38.8M 6s\n",
      " 15200K .......... .......... .......... .......... .......... 60% 12.3M 6s\n",
      " 15250K .......... .......... .......... .......... .......... 60% 2.15M 6s\n",
      " 15300K .......... .......... .......... .......... .......... 60%  528K 6s\n",
      " 15350K .......... .......... .......... .......... .......... 60% 6.73M 6s\n",
      " 15400K .......... .......... .......... .......... .......... 61%  110M 6s\n",
      " 15450K .......... .......... .......... .......... .......... 61% 19.5M 6s\n",
      " 15500K .......... .......... .......... .......... .......... 61% 2.14M 6s\n",
      " 15550K .......... .......... .......... .......... .......... 61%  528K 6s\n",
      " 15600K .......... .......... .......... .......... .......... 61% 6.48M 6s\n",
      " 15650K .......... .......... .......... .......... .......... 62% 51.5M 6s\n",
      " 15700K .......... .......... .......... .......... .......... 62%  114M 6s\n",
      " 15750K .......... .......... .......... .......... .......... 62% 2.36M 6s\n",
      " 15800K .......... .......... .......... .......... .......... 62%  530K 6s\n",
      " 15850K .......... .......... .......... .......... .......... 62% 4.16M 6s\n",
      " 15900K .......... .......... .......... .......... .......... 63% 70.9M 6s\n",
      " 15950K .......... .......... .......... .......... .......... 63% 50.3M 6s\n",
      " 16000K .......... .......... .......... .......... .......... 63% 2.44M 6s\n",
      " 16050K .......... .......... .......... .......... .......... 63%  531K 6s\n",
      " 16100K .......... .......... .......... .......... .......... 63% 4.72M 6s\n",
      " 16150K .......... .......... .......... .......... .......... 64% 20.3M 6s\n",
      " 16200K .......... .......... .......... .......... .......... 64% 28.9M 6s\n",
      " 16250K .......... .......... .......... .......... .......... 64% 54.9M 6s\n",
      " 16300K .......... .......... .......... .......... .......... 64%  476K 6s\n",
      " 16350K .......... .......... .......... .......... .......... 64% 2.72M 6s\n",
      " 16400K .......... .......... .......... .......... .......... 65% 20.8M 6s\n",
      " 16450K .......... .......... .......... .......... .......... 65% 38.4M 6s\n",
      " 16500K .......... .......... .......... .......... .......... 65% 33.5M 5s\n",
      " 16550K .......... .......... .......... .......... .......... 65%  914K 5s\n",
      " 16600K .......... .......... .......... .......... .......... 65%  766K 5s\n",
      " 16650K .......... .......... .......... .......... .......... 66% 9.17M 5s\n",
      " 16700K .......... .......... .......... .......... .......... 66% 70.4M 5s\n",
      " 16750K .......... .......... .......... .......... .......... 66% 18.1M 5s\n",
      " 16800K .......... .......... .......... .......... .......... 66% 2.35M 5s\n",
      " 16850K .......... .......... .......... .......... .......... 66%  513K 5s\n",
      " 16900K .......... .......... .......... .......... .......... 67% 8.14M 5s\n",
      " 16950K .......... .......... .......... .......... .......... 67% 44.1M 5s\n",
      " 17000K .......... .......... .......... .......... .......... 67% 16.1M 5s\n",
      " 17050K .......... .......... .......... .......... .......... 67% 2.90M 5s\n",
      " 17100K .......... .......... .......... .......... .......... 67%  532K 5s\n",
      " 17150K .......... .......... .......... .......... .......... 68% 4.43M 5s\n",
      " 17200K .......... .......... .......... .......... .......... 68% 17.9M 5s\n",
      " 17250K .......... .......... .......... .......... .......... 68% 17.0M 5s\n",
      " 17300K .......... .......... .......... .......... .......... 68% 35.4M 5s\n",
      " 17350K .......... .......... .......... .......... .......... 68%  490K 5s\n",
      " 17400K .......... .......... .......... .......... .......... 69% 2.72M 5s\n",
      " 17450K .......... .......... .......... .......... .......... 69% 14.4M 5s\n",
      " 17500K .......... .......... .......... .......... .......... 69% 35.5M 5s\n",
      " 17550K .......... .......... .......... .......... .......... 69% 16.3M 5s\n",
      " 17600K .......... .......... .......... .......... .......... 69%  967K 5s\n",
      " 17650K .......... .......... .......... .......... .......... 70%  772K 5s\n",
      " 17700K .......... .......... .......... .......... .......... 70% 7.62M 5s\n",
      " 17750K .......... .......... .......... .......... .......... 70% 24.5M 5s\n",
      " 17800K .......... .......... .......... .......... .......... 70% 15.2M 5s\n",
      " 17850K .......... .......... .......... .......... .......... 70% 2.99M 5s\n",
      " 17900K .......... .......... .......... .......... .......... 71%  515K 5s\n",
      " 17950K .......... .......... .......... .......... .......... 71% 5.63M 5s\n",
      " 18000K .......... .......... .......... .......... .......... 71% 16.3M 4s\n",
      " 18050K .......... .......... .......... .......... .......... 71% 59.2M 4s\n",
      " 18100K .......... .......... .......... .......... .......... 71% 15.1M 4s\n",
      " 18150K .......... .......... .......... .......... .......... 72%  499K 4s\n",
      " 18200K .......... .......... .......... .......... .......... 72% 2.69M 4s\n",
      " 18250K .......... .......... .......... .......... .......... 72% 14.5M 4s\n",
      " 18300K .......... .......... .......... .......... .......... 72% 18.4M 4s\n",
      " 18350K .......... .......... .......... .......... .......... 72% 14.8M 4s\n",
      " 18400K .......... .......... .......... .......... .......... 72% 1.39M 4s\n",
      " 18450K .......... .......... .......... .......... .......... 73%  628K 4s\n",
      " 18500K .......... .......... .......... .......... .......... 73% 7.11M 4s\n",
      " 18550K .......... .......... .......... .......... .......... 73% 20.1M 4s\n",
      " 18600K .......... .......... .......... .......... .......... 73% 19.7M 4s\n",
      " 18650K .......... .......... .......... .......... .......... 73% 2.87M 4s\n",
      " 18700K .......... .......... .......... .......... .......... 74%  534K 4s\n",
      " 18750K .......... .......... .......... .......... .......... 74% 5.43M 4s\n",
      " 18800K .......... .......... .......... .......... .......... 74% 11.6M 4s\n",
      " 18850K .......... .......... .......... .......... .......... 74% 22.7M 4s\n",
      " 18900K .......... .......... .......... .......... .......... 74% 13.0M 4s\n",
      " 18950K .......... .......... .......... .......... .......... 75%  998K 4s\n",
      " 19000K .......... .......... .......... .......... .......... 75%  776K 4s\n",
      " 19050K .......... .......... .......... .......... .......... 75% 6.50M 4s\n",
      " 19100K .......... .......... .......... .......... .......... 75% 31.0M 4s\n",
      " 19150K .......... .......... .......... .......... .......... 75% 19.4M 4s\n",
      " 19200K .......... .......... .......... .......... .......... 76% 2.86M 4s\n",
      " 19250K .......... .......... .......... .......... .......... 76%  530K 4s\n",
      " 19300K .......... .......... .......... .......... .......... 76% 4.17M 4s\n",
      " 19350K .......... .......... .......... .......... .......... 76% 22.7M 4s\n",
      " 19400K .......... .......... .......... .......... .......... 76% 32.2M 4s\n",
      " 19450K .......... .......... .......... .......... .......... 77% 12.6M 3s\n",
      " 19500K .......... .......... .......... .......... .......... 77%  555K 3s\n",
      " 19550K .......... .......... .......... .......... .......... 77% 2.00M 3s\n",
      " 19600K .......... .......... .......... .......... .......... 77% 6.86M 3s\n",
      " 19650K .......... .......... .......... .......... .......... 77% 24.7M 3s\n",
      " 19700K .......... .......... .......... .......... .......... 78% 33.6M 3s\n",
      " 19750K .......... .......... .......... .......... .......... 78% 2.84M 3s\n",
      " 19800K .......... .......... .......... .......... .......... 78%  530K 3s\n",
      " 19850K .......... .......... .......... .......... .......... 78% 6.11M 3s\n",
      " 19900K .......... .......... .......... .......... .......... 78% 7.70M 3s\n",
      " 19950K .......... .......... .......... .......... .......... 79% 47.5M 3s\n",
      " 20000K .......... .......... .......... .......... .......... 79% 11.6M 3s\n",
      " 20050K .......... .......... .......... .......... .......... 79% 1024K 3s\n",
      " 20100K .......... .......... .......... .......... .......... 79%  776K 3s\n",
      " 20150K .......... .......... .......... .......... .......... 79% 5.19M 3s\n",
      " 20200K .......... .......... .......... .......... .......... 80% 81.5M 3s\n",
      " 20250K .......... .......... .......... .......... .......... 80% 43.4M 3s\n",
      " 20300K .......... .......... .......... .......... .......... 80% 2.88M 3s\n",
      " 20350K .......... .......... .......... .......... .......... 80%  529K 3s\n",
      " 20400K .......... .......... .......... .......... .......... 80% 6.19M 3s\n",
      " 20450K .......... .......... .......... .......... .......... 81% 6.76M 3s\n",
      " 20500K .......... .......... .......... .......... .......... 81% 58.5M 3s\n",
      " 20550K .......... .......... .......... .......... .......... 81% 12.3M 3s\n",
      " 20600K .......... .......... .......... .......... .......... 81% 1.51M 3s\n",
      " 20650K .......... .......... .......... .......... .......... 81%  619K 3s\n",
      " 20700K .......... .......... .......... .......... .......... 82% 5.38M 3s\n",
      " 20750K .......... .......... .......... .......... .......... 82% 40.0M 3s\n",
      " 20800K .......... .......... .......... .......... .......... 82% 51.5M 3s\n",
      " 20850K .......... .......... .......... .......... .......... 82% 13.3M 3s\n",
      " 20900K .......... .......... .......... .......... .......... 82%  509K 3s\n",
      " 20950K .......... .......... .......... .......... .......... 83% 2.89M 3s\n",
      " 21000K .......... .......... .......... .......... .......... 83% 6.67M 3s\n",
      " 21050K .......... .......... .......... .......... .......... 83% 40.1M 2s\n",
      " 21100K .......... .......... .......... .......... .......... 83% 16.8M 2s\n",
      " 21150K .......... .......... .......... .......... .......... 83% 3.17M 2s\n",
      " 21200K .......... .......... .......... .......... .......... 84%  528K 2s\n",
      " 21250K .......... .......... .......... .......... .......... 84% 6.20M 2s\n",
      " 21300K .......... .......... .......... .......... .......... 84% 8.17M 2s\n",
      " 21350K .......... .......... .......... .......... .......... 84% 17.7M 2s\n",
      " 21400K .......... .......... .......... .......... .......... 84% 24.9M 2s\n",
      " 21450K .......... .......... .......... .......... .......... 85% 1.00M 2s\n",
      " 21500K .......... .......... .......... .......... .......... 85%  771K 2s\n",
      " 21550K .......... .......... .......... .......... .......... 85% 5.64M 2s\n",
      " 21600K .......... .......... .......... .......... .......... 85% 12.3M 2s\n",
      " 21650K .......... .......... .......... .......... .......... 85%  112M 2s\n",
      " 21700K .......... .......... .......... .......... .......... 86% 3.08M 2s\n",
      " 21750K .......... .......... .......... .......... .......... 86%  597K 2s\n",
      " 21800K .......... .......... .......... .......... .......... 86% 2.91M 2s\n",
      " 21850K .......... .......... .......... .......... .......... 86% 6.65M 2s\n",
      " 21900K .......... .......... .......... .......... .......... 86% 12.6M 2s\n",
      " 21950K .......... .......... .......... .......... .......... 87% 53.0M 2s\n",
      " 22000K .......... .......... .......... .......... .......... 87% 3.17M 2s\n",
      " 22050K .......... .......... .......... .......... .......... 87%  531K 2s\n",
      " 22100K .......... .......... .......... .......... .......... 87% 5.84M 2s\n",
      " 22150K .......... .......... .......... .......... .......... 87% 8.21M 2s\n",
      " 22200K .......... .......... .......... .......... .......... 88% 10.0M 2s\n",
      " 22250K .......... .......... .......... .......... .......... 88%  111M 2s\n",
      " 22300K .......... .......... .......... .......... .......... 88% 3.32M 2s\n",
      " 22350K .......... .......... .......... .......... .......... 88%  531K 2s\n",
      " 22400K .......... .......... .......... .......... .......... 88% 4.38M 2s\n",
      " 22450K .......... .......... .......... .......... .......... 89% 17.7M 2s\n",
      " 22500K .......... .......... .......... .......... .......... 89% 10.1M 2s\n",
      " 22550K .......... .......... .......... .......... .......... 89% 29.9M 2s\n",
      " 22600K .......... .......... .......... .......... .......... 89% 1.03M 2s\n",
      " 22650K .......... .......... .......... .......... .......... 89%  779K 2s\n",
      " 22700K .......... .......... .......... .......... .......... 90% 5.55M 1s\n",
      " 22750K .......... .......... .......... .......... .......... 90% 24.9M 1s\n",
      " 22800K .......... .......... .......... .......... .......... 90% 10.3M 1s\n",
      " 22850K .......... .......... .......... .......... .......... 90% 52.2M 1s\n",
      " 22900K .......... .......... .......... .......... .......... 90%  512K 1s\n",
      " 22950K .......... .......... .......... .......... .......... 90% 3.46M 1s\n",
      " 23000K .......... .......... .......... .......... .......... 91% 5.22M 1s\n",
      " 23050K .......... .......... .......... .......... .......... 91% 14.6M 1s\n",
      " 23100K .......... .......... .......... .......... .......... 91% 18.8M 1s\n",
      " 23150K .......... .......... .......... .......... .......... 91% 3.36M 1s\n",
      " 23200K .......... .......... .......... .......... .......... 91%  541K 1s\n",
      " 23250K .......... .......... .......... .......... .......... 92% 5.19M 1s\n",
      " 23300K .......... .......... .......... .......... .......... 92% 8.97M 1s\n",
      " 23350K .......... .......... .......... .......... .......... 92% 12.7M 1s\n",
      " 23400K .......... .......... .......... .......... .......... 92% 13.2M 1s\n",
      " 23450K .......... .......... .......... .......... .......... 92% 3.80M 1s\n",
      " 23500K .......... .......... .......... .......... .......... 93%  540K 1s\n",
      " 23550K .......... .......... .......... .......... .......... 93% 4.97M 1s\n",
      " 23600K .......... .......... .......... .......... .......... 93% 9.17M 1s\n",
      " 23650K .......... .......... .......... .......... .......... 93% 13.6M 1s\n",
      " 23700K .......... .......... .......... .......... .......... 93% 8.62M 1s\n",
      " 23750K .......... .......... .......... .......... .......... 94% 4.47M 1s\n",
      " 23800K .......... .......... .......... .......... .......... 94%  536K 1s\n",
      " 23850K .......... .......... .......... .......... .......... 94% 5.21M 1s\n",
      " 23900K .......... .......... .......... .......... .......... 94% 9.52M 1s\n",
      " 23950K .......... .......... .......... .......... .......... 94% 6.19M 1s\n",
      " 24000K .......... .......... .......... .......... .......... 95% 24.5M 1s\n",
      " 24050K .......... .......... .......... .......... .......... 95% 4.77M 1s\n",
      " 24100K .......... .......... .......... .......... .......... 95%  519K 1s\n",
      " 24150K .......... .......... .......... .......... .......... 95% 8.05M 1s\n",
      " 24200K .......... .......... .......... .......... .......... 95% 8.30M 1s\n",
      " 24250K .......... .......... .......... .......... .......... 96% 6.79M 1s\n",
      " 24300K .......... .......... .......... .......... .......... 96% 14.7M 1s\n",
      " 24350K .......... .......... .......... .......... .......... 96% 1.96M 1s\n",
      " 24400K .......... .......... .......... .......... .......... 96%  621K 0s\n",
      " 24450K .......... .......... .......... .......... .......... 96% 6.38M 0s\n",
      " 24500K .......... .......... .......... .......... .......... 97% 10.8M 0s\n",
      " 24550K .......... .......... .......... .......... .......... 97% 6.81M 0s\n",
      " 24600K .......... .......... .......... .......... .......... 97% 13.8M 0s\n",
      " 24650K .......... .......... .......... .......... .......... 97% 1.18M 0s\n",
      " 24700K .......... .......... .......... .......... .......... 97%  755K 0s\n",
      " 24750K .......... .......... .......... .......... .......... 98% 9.82M 0s\n",
      " 24800K .......... .......... .......... .......... .......... 98% 10.3M 0s\n",
      " 24850K .......... .......... .......... .......... .......... 98% 6.37M 0s\n",
      " 24900K .......... .......... .......... .......... .......... 98% 16.3M 0s\n",
      " 24950K .......... .......... .......... .......... .......... 98% 1.18M 0s\n",
      " 25000K .......... .......... .......... .......... .......... 99%  755K 0s\n",
      " 25050K .......... .......... .......... .......... .......... 99% 10.5M 0s\n",
      " 25100K .......... .......... .......... .......... .......... 99% 9.45M 0s\n",
      " 25150K .......... .......... .......... .......... .......... 99% 6.48M 0s\n",
      " 25200K .......... .......... .......... .......... .......... 99% 15.5M 0s\n",
      " 25250K .......... .......... .....                           100% 3.71M=14s\n",
      "\n",
      "2022-02-11 12:48:12 (1.72 MB/s) - ‘dbCAN-fam-V9.tar.gz’ saved [25882327/25882327]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "mkdir hh\n",
    "cd hh \n",
    "mkdir databases; cd databases\n",
    "wget http://wwwuser.gwdg.de/~compbiol/data/hhsuite/databases/hhsuite_dbs/dbCAN-fam-V9.tar.gz\n",
    "tar xzvf dbCAN-fam-V9.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KH9yJMqNgoOu"
   },
   "source": [
    "## Using hhsearch\n",
    "hhblits and hhsearch are the main functions in hhsuite which identify homologous proteins. They do this by calculating a profile hidden Markov model (HMM) from a given alignment and searching over a reference HMM proteome database using the Viterbi algorithm. Then the most similar HMMs are realigned and output to the user. To learn more, check out the original paper in the references above.\n",
    "\n",
    "Run a function from hhsuite with no parameters to read its documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FimKPlyq2rbU",
    "outputId": "0b20a394-b87d-4fbd-ef39-956624eda02b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HHsearch 3.3.0\n",
      "Search a database of HMMs with a query alignment or query HMM\n",
      "(c) The HH-suite development team\n",
      "Steinegger M, Meier M, Mirdita M, Vöhringer H, Haunsberger S J, and Söding J (2019)\n",
      "HH-suite3 for fast remote homology detection and deep protein annotation.\n",
      "BMC Bioinformatics, doi:10.1186/s12859-019-3019-7\n",
      "\n",
      "Usage: hhsearch -i query -d database [options]                       \n",
      " -i <file>      input/query multiple sequence alignment (a2m, a3m, FASTA) or HMM\n",
      "Options:                                                                        \n",
      " -d <name>      database name (e.g. uniprot20_29Feb2012)                        \n",
      "                Multiple databases may be specified with '-d <db1> -d <db2> ...'\n",
      " -e     [0,1]   E-value cutoff for inclusion in result alignment (def=0.001)       \n",
      "\n",
      "Input alignment format:                                                       \n",
      " -M a2m         use A2M/A3M (default): upper case = Match; lower case = Insert;\n",
      "               '-' = Delete; '.' = gaps aligned to inserts (may be omitted)   \n",
      " -M first       use FASTA: columns with residue in 1st sequence are match states\n",
      " -M [0,100]     use FASTA: columns with fewer than X% gaps are match states   \n",
      " -tags/-notags  do NOT / do neutralize His-, C-myc-, FLAG-tags, and trypsin \n",
      "                recognition sequence to background distribution (def=-notags)  \n",
      "\n",
      "Output options: \n",
      " -o <file>      write results in standard format to file (default=<infile.hhr>)\n",
      " -oa3m <file>   write result MSA with significant matches in a3m format\n",
      " -blasttab <name> write result in tabular BLAST format (compatible to -m 8 or -outfmt 6 output)\n",
      "                  1     2      3           4      5         6        7      8    9      10   11   12\n",
      "                  query target #match/tLen alnLen #mismatch #gapOpen qstart qend tstart tend eval score\n",
      " -add_cons      generate consensus sequence as master sequence of query MSA (default=don't)\n",
      " -hide_cons     don't show consensus sequence in alignments (default=show)     \n",
      " -hide_pred     don't show predicted 2ndary structure in alignments (default=show)\n",
      " -hide_dssp     don't show DSSP 2ndary structure in alignments (default=show)  \n",
      " -show_ssconf   show confidences for predicted 2ndary structure in alignments\n",
      "Filter options applied to query MSA, database MSAs, and result MSA              \n",
      " -all           show all sequences in result MSA; do not filter result MSA      \n",
      " -id   [0,100]  maximum pairwise sequence identity (def=90)\n",
      " -diff [0,inf[  filter MSAs by selecting most diverse set of sequences, keeping \n",
      "                at least this many seqs in each MSA block of length 50 \n",
      "                Zero and non-numerical values turn off the filtering. (def=100) \n",
      " -cov  [0,100]  minimum coverage with master sequence (%) (def=0)             \n",
      " -qid  [0,100]  minimum sequence identity with master sequence (%) (def=0)    \n",
      " -qsc  [0,100]  minimum score per column with master sequence (default=-20.0)    \n",
      " -neff [1,inf]  target diversity of multiple sequence alignment (default=off)   \n",
      " -mark          do not filter out sequences marked by \">@\"in their name line  \n",
      "\n",
      "HMM-HMM alignment options:                                                       \n",
      " -norealign          do NOT realign displayed hits with MAC algorithm (def=realign)   \n",
      " -ovlp <int>         banded alignment: forbid <ovlp> largest diagonals |i-j| of DP matrix (def=0)\n",
      " -mact [0,1[         posterior prob threshold for MAC realignment controlling greedi- \n",
      "                     ness at alignment ends: 0:global >0.1:local (default=0.35)       \n",
      " -glob/-loc          use global/local alignment mode for searching/ranking (def=local)\n",
      "Other options:                                                                   \n",
      " -v <int>       verbose mode: 0:no screen output  1:only warnings  2: verbose (def=2)\n",
      " -cpu <int>     number of CPUs to use (for shared memory SMPs) (default=2)      \n",
      "\n",
      "An extended list of options can be obtained by calling 'hhblits -h all'\n",
      "\n",
      "Example: hhsearch -i a.1.1.1.a3m -d scop70_1.71\n",
      "\n",
      "Download databases from <http://wwwuser.gwdg.de/~compbiol/data/hhsuite/databases/hhsuite_dbs/>.\n",
      "- 12:48:13.127 ERROR: Database is missing (see -d)!\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!hhsearch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mYV_ZrpTj8bj"
   },
   "source": [
    "Let's do an example. Say we have a protein which we want to compare to a MSA in order to identify any homologous regions. For this we can use hhsearch. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EsNXwU024Obb"
   },
   "source": [
    "Now let's take some protein sequence and search through the dbCAN database to see if we can find any potential homologous regions. First we will specify the sequence and save it as a FASTA file or a3m file in order to be readable by hhsearch. I pulled this sequence from the example query.a3m in the hhsuite data directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "id": "T8vVV57R4bWj"
   },
   "outputs": [],
   "source": [
    "with open('protein.fasta', 'w') as f:\n",
    "    f.write(\"\"\"\n",
    ">Uncharacterized bovine protein (Fragment)\n",
    "--PAGGQCtgiWHLLTRPLRP--QGRLPGLRVKYVFLVWLGVFAGSWMAYTHYSSYAELCRGHICQVVICDQFRKGIISGSICQDLCHLHQVEWRTCLSSVPGQQVYSGLWQGKEVTIKCGIEESLNSKAGSDGAPRRELVLFDKPSRGTSIKEFREMTLSFLKANLGDLPSLPALVGRVLLMADFNKDNRVSLAEAKSVWALLQRNEFLLLLSLQEKEHASRLLGYCGDLYVTEGVPLSSWPGATLPPLLRPLLPPALHGALQQWLGPAWPWRAKIAMGLLEFVEDLFHGAYGNFYMCETTLANVGYTAKYDFRMADLQQVAPEAAVRRFLRGRRCEHSADCTYGRDCRAPCDTLMRQCKGDLVQPNLAKVCELLRDYLLPGAPAALRPELGKQLRTCTTLSGLASQVEAHHSLVLSHLKSLLWKEISDSRYT\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yvpaiykr4dvG"
   },
   "source": [
    "Then we can call hhsearch, specifying the query sequence with the -i flag, the database to search through with -d, and the output with -o."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "rXAkz1UzhoK4",
    "outputId": "717b99b5-1e3e-4bef-bb38-5a90cf3d733e"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "- 12:48:13.301 INFO: Search results will be written to /home/tony/github/deepchem/examples/tutorials/protein.hhr\n",
      "\n",
      "- 12:48:13.331 INFO: /home/tony/github/deepchem/examples/tutorials/protein.fasta is in A2M, A3M or FASTA format\n",
      "\n",
      "- 12:48:13.331 WARNING: Input alignment /home/tony/github/deepchem/examples/tutorials/protein.fasta looks like aligned FASTA instead of A2M/A3M format. Consider using '-M first' or '-M 50'\n",
      "\n",
      "- 12:48:13.331 INFO: NOTE: Use the '-add_cons' option to calculate a consensus sequence as first sequence of the alignment with hhconsensus or hhmake.\n",
      "\n",
      "- 12:48:13.331 INFO: Searching 683 database HHMs without prefiltering\n",
      "\n",
      "- 12:48:13.332 INFO: Iteration 1\n",
      "\n",
      "- 12:48:13.420 INFO: Scoring 683 HMMs using HMM-HMM Viterbi alignment\n",
      "\n",
      "- 12:48:13.460 INFO: Alternative alignment: 0\n",
      "\n",
      "- 12:48:13.611 INFO: 683 alignments done\n",
      "\n",
      "- 12:48:13.612 INFO: Alternative alignment: 1\n",
      "\n",
      "- 12:48:13.625 INFO: 38 alignments done\n",
      "\n",
      "- 12:48:13.625 INFO: Alternative alignment: 2\n",
      "\n",
      "- 12:48:13.629 INFO: 3 alignments done\n",
      "\n",
      "- 12:48:13.629 INFO: Alternative alignment: 3\n",
      "\n",
      "- 12:48:13.655 INFO: Premerge done\n",
      "\n",
      "- 12:48:13.656 INFO: Realigning 10 HMM-HMM alignments using Maximum Accuracy algorithm\n",
      "\n",
      "- 12:48:13.692 INFO: 0 sequences belonging to 0 database HMMs found with an E-value < 0.001\n",
      "\n",
      "- 12:48:13.692 INFO: Number of effective sequences of resulting query HMM: Neff = 1\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from deepchem.utils import sequence_utils\n",
    "dataset_path = 'protein.fasta'\n",
    "data_dir = 'hh/databases'\n",
    "results = sequence_utils.hhsearch(dataset_path,database='dbCAN-fam-V9', data_dir=data_dir)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query         Uncharacterized bovine protein (Fragment)\n",
      "Match_columns 431\n",
      "No_of_seqs    1 out of 1\n",
      "Neff          1\n",
      "Searched_HMMs 683\n",
      "Date          Fri Feb 11 12:48:13 2022\n",
      "Command       hhsearch -i /home/tony/github/deepchem/examples/tutorials/protein.fasta -d hh/databases/dbCAN-fam-V9 -oa3m /home/tony/github/deepchem/examples/tutorials/results.a3m -cpu 4 -e 0.001 \n",
      "\n",
      " No Hit                             Prob E-value P-value  Score    SS Cols Query HMM  Template HMM\n",
      "  1 ABJ15796.1|231-344|9.6e-33       8.2     2.9  0.0042   25.2   0.0   13  224-236    40-52  (116)\n",
      "  2 lcl|consensus                    5.1     5.2  0.0076   17.1   0.0   14  182-195     1-14  (21)\n",
      "  3 ABW08129.1|GT4|GT97||563-891     4.8     5.7  0.0084   26.6   0.0   46  104-150    93-140 (329)\n",
      "  4 AEO62162.1|AA13||19-250          4.6       6  0.0087   25.5   0.0   18  330-347   139-156 (232)\n",
      "  5 BAF49076.1|GH5_26.hmm|8.3e-11|   2.4      13    0.02   21.9   0.0   12  287-298    45-56  (141)\n",
      "  6 BBD44721.1 Hypothetical protei   2.3      14    0.02   25.7   0.0   81  110-221   326-406 (552)\n",
      "  7 AAU92474.1|CBM2|2-82|1.9e-23     2.3      14    0.02   19.1   0.0   19  222-240    13-33  (104)\n",
      "  8 BAX82587.1 hypothetical protei   2.3      14   0.021   25.7   0.0   25  104-128   466-490 (656)\n",
      "  9 AHE46274.1|GH13_13.hmm|1.6e-20   2.0      16   0.024   24.1   0.0   45  143-199    99-143 (393)\n",
      " 10 ACF55060.1|GH13_13.hmm|2.5e-47   1.9      17   0.025   23.2   0.0   22  144-165    74-95  (330)\n",
      "\n",
      "No 1\n",
      ">ABJ15796.1|231-344|9.6e-33\n",
      "Probab=8.16  E-value=2.9  Score=25.22  Aligned_cols=13  Identities=46%  Similarity=0.795  Sum_probs=10.2  Template_Neff=3.400\n",
      "\n",
      "Q Uncharacterize  223 YCGDLYVTEGVPL  235 (430)\n",
      "Q Consensus       224 ycgdlyvtegvpl  236 (431)\n",
      "                      --||||.||||--\n",
      "T Consensus        40 I~Gnlyi~eGVG~   52 (116)\n",
      "T ABJ15796.1|231   40 INGNLYIAEGVGE   52 (116)\n",
      "Confidence            3599999999853\n",
      "\n",
      "\n",
      "No 2\n",
      ">lcl|consensus\n",
      "Probab=5.13  E-value=5.2  Score=17.13  Aligned_cols=14  Identities=29%  Similarity=0.437  Sum_probs=10.2  Template_Neff=4.300\n",
      "\n",
      "Q Uncharacterize  181 DFNKDNRVSLAEAK  194 (430)\n",
      "Q Consensus       182 dfnkdnrvslaeak  195 (431)\n",
      "                      |.|.|++|+-.++-\n",
      "T Consensus         1 DvN~DG~Vna~D~~   14 (21)\n",
      "T lcl|consensus_    1 DVNGDGKVNALDLA   14 (21)\n",
      "Confidence            67888888766553\n",
      "\n",
      "\n",
      "No 3\n",
      ">ABW08129.1|GT4|GT97||563-891\n",
      "Probab=4.78  E-value=5.7  Score=26.58  Aligned_cols=46  Identities=20%  Similarity=0.367  Sum_probs=28.5  Template_Neff=1.500\n",
      "\n",
      "Q Uncharacterize  103 YSGLWQGKEVTIKCGIEESLN--SKAGSDGAPRRELVLFDKPSRGTSIK  149 (430)\n",
      "Q Consensus       104 ysglwqgkevtikcgieesln--skagsdgaprrelvlfdkpsrgtsik  150 (431)\n",
      "                      ..|+|. |...+.-.|.-...  .+..-|.+|..|-++||-|.||-+-+\n",
      "T Consensus        93 ~~G~W~-~~~~~~~~i~~~~DheG~r~m~~~~~~~T~i~e~~Rk~~~~~  140 (329)\n",
      "T ABW08129.1|GT4   93 FTGKWE-KHFQTSPKIDYRFDHEGKRSMDDVFSEETFIMEFPRKNGIDK  140 (329)\n",
      "Confidence            457774 33333333433333  45556778888889999988887654\n",
      "\n",
      "\n",
      "No 4\n",
      ">AEO62162.1|AA13||19-250\n",
      "Probab=4.61  E-value=6  Score=25.50  Aligned_cols=18  Identities=39%  Similarity=0.936  Sum_probs=14.8  Template_Neff=1.600\n",
      "\n",
      "Q Uncharacterize  329 RGRRCEHSADCTYGRDCR  346 (430)\n",
      "Q Consensus       330 rgrrcehsadctygrdcr  347 (431)\n",
      "                      .|..|..|+||+-|..|-\n",
      "T Consensus       139 ~Gq~C~y~pDC~~gq~C~  156 (232)\n",
      "T AEO62162.1|AA1  139 SGQTCGYSPDCSPGQPCW  156 (232)\n",
      "Confidence            467899999999998774\n",
      "\n",
      "\n",
      "No 5\n",
      ">BAF49076.1|GH5_26.hmm|8.3e-11|182-335\n",
      "Probab=2.39  E-value=13  Score=21.92  Aligned_cols=12  Identities=33%  Similarity=0.720  Sum_probs=9.5  Template_Neff=1.900\n",
      "\n",
      "Q Uncharacterize  286 HGAYGNFYMCET  297 (430)\n",
      "Q Consensus       287 hgaygnfymcet  298 (431)\n",
      "                      .|+|++|||-..\n",
      "T Consensus        45 ~G~yn~~Y~l~s   56 (141)\n",
      "T BAF49076.1|GH5   45 QGTYNGNYMLTS   56 (141)\n",
      "Confidence            478999998654\n",
      "\n",
      "\n",
      "No 6\n",
      ">BBD44721.1 Hypothetical protein PEIBARAKI_4714 [Petrimonas sp. IBARAKI]\n",
      "Probab=2.34  E-value=14  Score=25.75  Aligned_cols=81  Identities=23%  Similarity=0.240  Sum_probs=46.6  Template_Neff=3.400\n",
      "\n",
      "Q Uncharacterize  109 GKEVTIKCGIEESLNSKAGSDGAPRRELVLFDKPSRGTSIKEFREMTLSFLKANLGDLPSLPALVGRVLLMADFNKDNRV  188 (430)\n",
      "Q Consensus       110 gkevtikcgieeslnskagsdgaprrelvlfdkpsrgtsikefremtlsflkanlgdlpslpalvgrvllmadfnkdnrv  189 (431)\n",
      "                      |..|+|-.|+|+..+-+.             +++-.-.|+.-+|-+.+++|.+--.          .|-+   ||-.+-+\n",
      "T Consensus       326 g~~V~Iya~l~~~~~~~~-------------~~~~~~~S~~~~Rg~Aa~~L~rGAd----------GIyl---FN~f~~~  379 (552)\n",
      "T BBD44721.1_con  326 GTGVKIYAGLEDARAPDP-------------STRRETNSLEAYRGRAANALSRGAD----------GIYL---FNYFYPP  379 (552)\n",
      "Confidence            777888888888744332             4455566888888888888765322          2333   4443332\n",
      "\n",
      "\n",
      "Q Uncharacterize  189 SLAEAKSVWALLQRNEFLLLLSLQEKEHASRL  220 (430)\n",
      "Q Consensus       190 slaeaksvwallqrnefllllslqekehasrl  221 (431)\n",
      "                      ..     --.|||.-.=+-.|.-|+|.|+-..\n",
      "T Consensus       380 ~~-----~~~llrelgd~~~L~~~~K~y~~s~  406 (552)\n",
      "T BBD44721.1_con  380 QM-----RSPLLRELGDLETLATQEKLYALSI  406 (552)\n",
      "Confidence            11     1233443333445566788876543\n",
      "\n",
      "\n",
      "No 7\n",
      ">AAU92474.1|CBM2|2-82|1.9e-23\n",
      "Probab=2.33  E-value=14  Score=19.07  Aligned_cols=19  Identities=26%  Similarity=0.562  Sum_probs=14.3  Template_Neff=6.600\n",
      "\n",
      "Q Uncharacterize  221 LGYCGDLYVT--EGVPLSSWP  239 (430)\n",
      "Q Consensus       222 lgycgdlyvt--egvplsswp  240 (431)\n",
      "                      -||++++-|+  ...+++.|-\n",
      "T Consensus        13 ~Gf~~~v~vtN~~~~~i~~W~   33 (104)\n",
      "T AAU92474.1|CBM   13 GGFQANVTVTNTGSSAISGWT   33 (104)\n",
      "Confidence            3789998888  567777774\n",
      "\n",
      "\n",
      "No 8\n",
      ">BAX82587.1 hypothetical protein ALGA_4297 [Marinifilaceae bacterium SPP2]\n",
      "Probab=2.28  E-value=14  Score=25.65  Aligned_cols=25  Identities=40%  Similarity=0.314  Sum_probs=21.4  Template_Neff=1.500\n",
      "\n",
      "Q Uncharacterize  103 YSGLWQGKEVTIKCGIEESLNSKAG  127 (430)\n",
      "Q Consensus       104 ysglwqgkevtikcgieeslnskag  128 (431)\n",
      "                      -+.+||+|.+.||..||.|-|-+--\n",
      "T Consensus       466 ~kd~~~tk~~sik~kietSenFtl~  490 (656)\n",
      "T BAX82587.1_con  466 NKDLNQTKQVSIKTKIETSENFTLS  490 (656)\n",
      "Confidence            5789999999999999998876543\n",
      "\n",
      "\n",
      "No 9\n",
      ">AHE46274.1|GH13_13.hmm|1.6e-201|415-835\n",
      "Probab=2.04  E-value=16  Score=24.14  Aligned_cols=45  Identities=29%  Similarity=0.332  Sum_probs=25.7  Template_Neff=3.900\n",
      "\n",
      "Q Uncharacterize  142 PSRGTSIKEFREMTLSFLKANLGDLPSLPALVGRVLLMADFNKDNRVSLAEAKSVWA  198 (430)\n",
      "Q Consensus       143 psrgtsikefremtlsflkanlgdlpslpalvgrvllmadfnkdnrvslaeaksvwa  199 (431)\n",
      "                      |.-.+-|+|||+|.-+.           -+.=-||.+=.=||--+-.-+ .++||..\n",
      "T Consensus        99 ~~g~~Ri~EfR~MV~al-----------h~~GlrVv~DVVyNHT~~sg~-~~~SVlD  143 (393)\n",
      "T AHE46274.1|GH1   99 PDGVARIKEFRAMVQAL-----------HAMGLRVVMDVVYNHTAASGQ-YDNSVLD  143 (393)\n",
      "Confidence            34445589999998653           222235655555676655544 3345543\n",
      "\n",
      "\n",
      "No 10\n",
      ">ACF55060.1|GH13_13.hmm|2.5e-47|336-542\n",
      "Probab=1.94  E-value=17  Score=23.24  Aligned_cols=22  Identities=23%  Similarity=0.371  Sum_probs=17.6  Template_Neff=4.500\n",
      "\n",
      "Q Uncharacterize  143 SRGTSIKEFREMTLSFLKANLG  164 (430)\n",
      "Q Consensus       144 srgtsikefremtlsflkanlg  165 (431)\n",
      "                      +.-+.|+||++|...+=++.++\n",
      "T Consensus        74 dp~~RI~E~K~mI~~lH~~GI~   95 (330)\n",
      "T ACF55060.1|GH1   74 DPYGRIREFKQMIQALHDAGIR   95 (330)\n",
      "Confidence            4456799999999988887765\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "#open the results and print them\n",
    "f = open(\"protein.hhr\", \"r\")\n",
    "print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NVu-UTBGbi52"
   },
   "source": [
    "Two files are output and saved to the dataset directory, results.hhr and results.a3m. results.hhr is the hhsuite results file, which is a summary of the results. results.a3m is the actual MSA file.\n",
    "\n",
    "In the hhr file, the 'Prob' column describes the estimated probability of the query sequence being at least partially homologous to the template. Probabilities of 95% or more are nearly certain, and probabilities of 30% or more call for closer consideration. The E value tells you how many random matches with a better score would be expected if the searched database was unrelated to the query sequence. These results show that none of the sequences align well with our randomly chosen protein, which is to be expected because our query sequence was chosen at random.\n",
    "\n",
    "Now let's check the results if we use a sequence that we know will align with something in the dbCAN database. I pulled this protein from the dockerin.faa file in dbCAN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "EPHzIdM2e52D",
    "outputId": "d5b1a0ba-510d-449e-9df2-0d9cd7da0750"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "- 12:48:13.823 INFO: Search results will be written to /home/tony/github/deepchem/examples/tutorials/protein2.hhr\n",
      "\n",
      "- 12:48:13.851 INFO: /home/tony/github/deepchem/examples/tutorials/protein2.fasta is in A2M, A3M or FASTA format\n",
      "\n",
      "- 12:48:13.852 INFO: Searching 683 database HHMs without prefiltering\n",
      "\n",
      "- 12:48:13.852 INFO: Iteration 1\n",
      "\n",
      "- 12:48:13.873 INFO: Scoring 683 HMMs using HMM-HMM Viterbi alignment\n",
      "\n",
      "- 12:48:13.913 INFO: Alternative alignment: 0\n",
      "\n",
      "- 12:48:13.979 INFO: 683 alignments done\n",
      "\n",
      "- 12:48:13.979 INFO: Alternative alignment: 1\n",
      "\n",
      "- 12:48:13.982 INFO: 10 alignments done\n",
      "\n",
      "- 12:48:13.982 INFO: Alternative alignment: 2\n",
      "\n",
      "- 12:48:13.984 INFO: 3 alignments done\n",
      "\n",
      "- 12:48:13.984 INFO: Alternative alignment: 3\n",
      "\n",
      "- 12:48:13.986 INFO: 3 alignments done\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query         dockerin,22,NCBI-Bacteria,gi|125972715|ref|YP_001036625.1|,162-245,0.033\n",
      "Match_columns 84\n",
      "No_of_seqs    1 out of 1\n",
      "Neff          1\n",
      "Searched_HMMs 683\n",
      "Date          Fri Feb 11 12:48:14 2022\n",
      "Command       hhsearch -i /home/tony/github/deepchem/examples/tutorials/protein2.fasta -d hh/databases/dbCAN-fam-V9 -oa3m /home/tony/github/deepchem/examples/tutorials/results.a3m -cpu 4 -e 0.001 \n",
      "\n",
      " No Hit                             Prob E-value P-value  Score    SS Cols Query HMM  Template HMM\n",
      "  1 lcl|consensus                   97.0 5.9E-08 8.7E-11   43.5   0.0   21    4-24      1-21  (21)\n",
      "  2 ABN51673.1|GH124|2-334|2.6e-21  92.5 0.00033 4.8E-07   45.5   0.0   68    1-75     21-88  (318)\n",
      "  3 AAK20911.1|PL11|47-657|0        15.7     1.1  0.0017   27.6   0.0   14    1-14    329-342 (606)\n",
      "  4 AGE62576.1|PL11_1.hmm|0|1-596   10.2     2.1  0.0031   26.0   0.0   13    1-13    118-130 (602)\n",
      "  5 AAZ21803.1|GH103|26-328|1.7e-8   9.3     2.4  0.0035   22.4   0.0   10    4-13    175-184 (293)\n",
      "  6 AGE62576.1|PL11_1.hmm|0|1-596    5.5     4.8   0.007   23.9   0.0   12    1-12    329-340 (602)\n",
      "  7 AAK20911.1|PL11|47-657|0         5.5     4.8   0.007   23.8   0.0   13    1-13    118-130 (606)\n",
      "  8 APU21542.1|PL11_2.hmm|1.4e-162   4.9     5.6  0.0082   23.5   0.0   14    2-15    318-331 (579)\n",
      "  9 AAK20911.1|PL11|47-657|0         4.7     5.8  0.0084   23.4   0.0   10    3-12    184-193 (606)\n",
      " 10 AGE62576.1|PL11_1.hmm|0|1-596    4.6       6  0.0088   23.3   0.0    7    4-10    185-191 (602)\n",
      "\n",
      "No 1\n",
      ">lcl|consensus\n",
      "Probab=97.03  E-value=5.9e-08  Score=43.48  Aligned_cols=21  Identities=57%  Similarity=1.061  Sum_probs=20.1  Template_Neff=4.300\n",
      "\n",
      "Q dockerin,22,NC    4 DLNGDGKITSSDYNLLKRYIL   24 (84)\n",
      "Q Consensus         4 dlngdgkitssdynllkryil   24 (84)\n",
      "                      |+|+||+|+..|+.++|||+|\n",
      "T Consensus         1 DvN~DG~Vna~D~~~l~~~l~   21 (21)\n",
      "T lcl|consensus_    1 DVNGDGKVNALDLALLKKYLL   21 (21)\n",
      "Confidence            899999999999999999986\n",
      "\n",
      "\n",
      "No 2\n",
      ">ABN51673.1|GH124|2-334|2.6e-219\n",
      "Probab=92.52  E-value=0.00033  Score=45.54  Aligned_cols=68  Identities=31%  Similarity=0.523  Sum_probs=51.6  Template_Neff=1.400\n",
      "\n",
      "Q dockerin,22,NC    1 SCADLNGDGKITSSDYNLLKRYILHLIDKFPIGNDETDEGINDGFNDETDEDINDSFIEANSKFAFDIFKQISKD   75 (84)\n",
      "Q Consensus         1 scadlngdgkitssdynllkryilhlidkfpigndetdegindgfndetdedindsfieanskfafdifkqiskd   75 (84)\n",
      "                      ++||+||||+|+||||+|||| ||++|++||+++|||+|++|..      .-|+|.--.+-.++.....+.+.|.\n",
      "T Consensus        21 v~GD~n~dgvv~isd~vl~k~-~l~~~a~~~a~~d~w~g~vN~d------d~I~D~d~~~~kryll~mir~~pk~   88 (318)\n",
      "T ABN51673.1|GH1   21 VIGDVNADGVVNISDYVLMKR-ILRIIADFPADDDMWVGDVNGD------DVINDIDCNYLKRYLLHMIREFPKN   88 (318)\n",
      "Confidence            489999999999999999999 9999999999999999999854      3333333333344444444444443\n",
      "\n",
      "\n",
      "No 3\n",
      ">AAK20911.1|PL11|47-657|0\n",
      "Probab=15.69  E-value=1.1  Score=27.56  Aligned_cols=14  Identities=50%  Similarity=0.641  Sum_probs=10.4  Template_Neff=3.500\n",
      "\n",
      "Q dockerin,22,NC    1 SCADLNGDGKITSS   14 (84)\n",
      "Q Consensus         1 scadlngdgkitss   14 (84)\n",
      "                      |++|+|+||+=.|-\n",
      "T Consensus       329 svaDVDgDGkDEIi  342 (606)\n",
      "T AAK20911.1|PL1  329 SVADVDGDGKDEII  342 (606)\n",
      "Confidence            57889998886553\n",
      "\n",
      "\n",
      "No 4\n",
      ">AGE62576.1|PL11_1.hmm|0|1-596\n",
      "Probab=10.22  E-value=2.1  Score=26.01  Aligned_cols=13  Identities=46%  Similarity=0.772  Sum_probs=10.8  Template_Neff=3.300\n",
      "\n",
      "Q dockerin,22,NC    1 SCADLNGDGKITS   13 (84)\n",
      "Q Consensus         1 scadlngdgkits   13 (84)\n",
      "                      |+||||+||...|\n",
      "T Consensus       118 SVGDLDGDG~YEi  130 (602)\n",
      "T AGE62576.1|PL1  118 SVGDLDGDGEYEI  130 (602)\n",
      "Confidence            6899999998654\n",
      "\n",
      "\n",
      "No 5\n",
      ">AAZ21803.1|GH103|26-328|1.7e-83\n",
      "Probab=9.26  E-value=2.4  Score=22.41  Aligned_cols=10  Identities=40%  Similarity=0.833  Sum_probs=9.2  Template_Neff=5.600\n",
      "\n",
      "Q dockerin,22,NC    4 DLNGDGKITS   13 (84)\n",
      "Q Consensus         4 dlngdgkits   13 (84)\n",
      "                      |.|+||++++\n",
      "T Consensus       175 D~DgDG~~Dl  184 (293)\n",
      "T AAZ21803.1|GH1  175 DFDGDGRRDL  184 (293)\n",
      "Confidence            8899999997\n",
      "\n",
      "\n",
      "No 6\n",
      ">AGE62576.1|PL11_1.hmm|0|1-596\n",
      "Probab=5.50  E-value=4.8  Score=23.90  Aligned_cols=12  Identities=58%  Similarity=0.847  Sum_probs=7.5  Template_Neff=3.300\n",
      "\n",
      "Q dockerin,22,NC    1 SCADLNGDGKIT   12 (84)\n",
      "Q Consensus         1 scadlngdgkit   12 (84)\n",
      "                      |+||+|+||+=.\n",
      "T Consensus       329 svaDVDgDG~DE  340 (602)\n",
      "T AGE62576.1|PL1  329 SVADVDGDGKDE  340 (602)\n",
      "Confidence            467777777633\n",
      "\n",
      "\n",
      "No 7\n",
      ">AAK20911.1|PL11|47-657|0\n",
      "Probab=5.47  E-value=4.8  Score=23.84  Aligned_cols=13  Identities=46%  Similarity=0.772  Sum_probs=10.6  Template_Neff=3.500\n",
      "\n",
      "Q dockerin,22,NC    1 SCADLNGDGKITS   13 (84)\n",
      "Q Consensus         1 scadlngdgkits   13 (84)\n",
      "                      |+||||+||...+\n",
      "T Consensus       118 SVGDLDGDG~yEi  130 (606)\n",
      "T AAK20911.1|PL1  118 SVGDLDGDGEYEI  130 (606)\n",
      "Confidence            6899999998654\n",
      "\n",
      "\n",
      "No 8\n",
      ">APU21542.1|PL11_2.hmm|1.4e-162|44-417\n",
      "Probab=4.86  E-value=5.6  Score=23.51  Aligned_cols=14  Identities=50%  Similarity=0.715  Sum_probs=9.4  Template_Neff=2.600\n",
      "\n",
      "Q dockerin,22,NC    2 CADLNGDGKITSSD   15 (84)\n",
      "Q Consensus         2 cadlngdgkitssd   15 (84)\n",
      "                      +.|+|+||+=.+++\n",
      "T Consensus       318 ~~DvD~DG~DEi~~  331 (579)\n",
      "T APU21542.1|PL1  318 IVDVDGDGKDEISD  331 (579)\n",
      "Confidence            45777777766655\n",
      "\n",
      "\n",
      "No 9\n",
      ">AAK20911.1|PL11|47-657|0\n",
      "Probab=4.74  E-value=5.8  Score=23.38  Aligned_cols=10  Identities=50%  Similarity=0.896  Sum_probs=5.6  Template_Neff=3.500\n",
      "\n",
      "Q dockerin,22,NC    3 ADLNGDGKIT   12 (84)\n",
      "Q Consensus         3 adlngdgkit   12 (84)\n",
      "                      -|+|+|||-.\n",
      "T Consensus       184 yD~DGDGkAE  193 (606)\n",
      "T AAK20911.1|PL1  184 YDFDGDGKAE  193 (606)\n",
      "Confidence            3666666543\n",
      "\n",
      "\n",
      "No 10\n",
      ">AGE62576.1|PL11_1.hmm|0|1-596\n",
      "Probab=4.58  E-value=6  Score=23.30  Aligned_cols=7  Identities=71%  Similarity=1.426  Sum_probs=0.0  Template_Neff=3.300\n",
      "\n",
      "Q dockerin,22,NC    4 DLNGDGK   10 (84)\n",
      "Q Consensus         4 dlngdgk   10 (84)\n",
      "                      |||||||\n",
      "T Consensus       185 D~DGDGk  191 (602)\n",
      "T AGE62576.1|PL1  185 DFDGDGK  191 (602)\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "- 12:48:14.063 INFO: Premerge done\n",
      "\n",
      "- 12:48:14.063 INFO: Realigning 10 HMM-HMM alignments using Maximum Accuracy algorithm\n",
      "\n",
      "- 12:48:14.084 INFO: 4 sequences belonging to 4 database HMMs found with an E-value < 0.001\n",
      "\n",
      "- 12:48:14.084 INFO: Number of effective sequences of resulting query HMM: Neff = 1.39047\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open('protein2.fasta', 'w') as f:\n",
    "    f.write(\"\"\">dockerin,22,NCBI-Bacteria,gi|125972715|ref|YP_001036625.1|,162-245,0.033\n",
    "SCADLNGDGKITSSDYNLLKRYILHLIDKFPIGNDETDEGINDGFNDETDEDINDSFIEANSKFAFDIFKQISKDEQGKNVFIS\n",
    "\"\"\")\n",
    "    \n",
    "dataset_path = 'protein2.fasta'\n",
    "sequence_utils.hhsearch(dataset_path,database='dbCAN-fam-V9', data_dir=data_dir)\n",
    "\n",
    "#open the results and print them\n",
    "f = open(\"protein2.hhr\", \"r\")\n",
    "print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Gn8xxUpifxLJ"
   },
   "source": [
    "As you can see, there are 2 sequences which are a match for our query sequence. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "t66in632eD5W"
   },
   "source": [
    "## Using hhblits\n",
    "hhblits works in much the same way as hhsearch, but it is much faster and slightly less sensitive. This would be more suited to searching very large databases, or producing a MSA with multiple sequences instead of just one. Let's make use of that by using our query sequence to create an MSA. We could then use that MSA, with its family of proteins, to search a larger database for potential matches. This will be much more effective than searching a large database with a single sequence.\n",
    "\n",
    "We will use the same dbCAN database. I will pull a glycoside hydrolase protein from UnipProt, so it will likely be related to some proteins in dbCAN, which has carbohydrate-active enzymes.\n",
    "\n",
    "The option -oa3m will tell hhblits to output an MSA as an a3m file. The -n option specifies the number of iterations. This is recommended to keep between 1 and 4, we will try 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "cHLgNXnalhsG",
    "outputId": "301fb830-22ef-4c40-a3bc-b0b7918405fe"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2022-02-11 12:48:14--  https://www.uniprot.org/uniprot/G8M3C3.fasta\n",
      "Resolving www.uniprot.org (www.uniprot.org)... 193.62.193.81\n",
      "Connecting to www.uniprot.org (www.uniprot.org)|193.62.193.81|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 \n",
      "Length: 897 [text/plain]\n",
      "Saving to: ‘protein3.fasta’\n",
      "\n",
      "protein3.fasta      100%[===================>]     897  --.-KB/s    in 0s      \n",
      "\n",
      "2022-02-11 12:48:15 (1.70 GB/s) - ‘protein3.fasta’ saved [897/897]\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "- 12:48:15.242 WARNING: Ignoring unknown option -n\n",
      "\n",
      "- 12:48:15.242 WARNING: Ignoring unknown option 2\n",
      "\n",
      "- 12:48:15.242 INFO: Search results will be written to /home/tony/github/deepchem/examples/tutorials/protein3.hhr\n",
      "\n",
      "- 12:48:15.270 INFO: /home/tony/github/deepchem/examples/tutorials/protein3.fasta is in A2M, A3M or FASTA format\n",
      "\n",
      "- 12:48:15.271 INFO: Searching 683 database HHMs without prefiltering\n",
      "\n",
      "- 12:48:15.271 INFO: Iteration 1\n",
      "\n",
      "- 12:48:15.424 INFO: Scoring 683 HMMs using HMM-HMM Viterbi alignment\n",
      "\n",
      "- 12:48:15.465 INFO: Alternative alignment: 0\n",
      "\n",
      "- 12:48:15.658 INFO: 683 alignments done\n",
      "\n",
      "- 12:48:15.659 INFO: Alternative alignment: 1\n",
      "\n",
      "- 12:48:15.697 INFO: 92 alignments done\n",
      "\n",
      "- 12:48:15.697 INFO: Alternative alignment: 2\n",
      "\n",
      "- 12:48:15.710 INFO: 7 alignments done\n",
      "\n",
      "- 12:48:15.710 INFO: Alternative alignment: 3\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query         tr|G8M3C3|G8M3C3_HUNCD Dockerin-like protein OS=Hungateiclostridium clariflavum (strain DSM 19732 / NBRC 101661 / EBR45) OX=720554 GN=Clocl_4007 PE=4 SV=1\n",
      "Match_columns 728\n",
      "No_of_seqs    1 out of 1\n",
      "Neff          1\n",
      "Searched_HMMs 683\n",
      "Date          Fri Feb 11 12:48:16 2022\n",
      "Command       hhsearch -i /home/tony/github/deepchem/examples/tutorials/protein3.fasta -d hh/databases/dbCAN-fam-V9 -oa3m /home/tony/github/deepchem/examples/tutorials/results.a3m -cpu 4 -n 2 -e 0.001 \n",
      "\n",
      " No Hit                             Prob E-value P-value  Score    SS Cols Query HMM  Template HMM\n",
      "  1 AAA91086.1|GH48|150-238|4.7e-1 100.0  7E-195  1E-197 1475.1   0.0  608   31-644     1-619 (620)\n",
      "  2 lcl|consensus                   91.8 0.00051 7.4E-07   37.5   0.0   20  668-687     1-20  (21)\n",
      "  3 ABN51673.1|GH124|2-334|2.6e-21  52.5   0.096 0.00014   40.1   0.0   66  663-728    19-85  (318)\n",
      "  4 CAR68154.1|GH88|62-388|4.9e-13  10.5       2   0.003   30.7   0.0   43  421-463   181-223 (329)\n",
      "  5 ACY49347.1|GH105|46-385|1.1e-1   6.4       4  0.0058   28.2   0.0   60  324-383   169-228 (329)\n",
      "  6 QGI59602.1|GH16_22|78-291        5.4     4.9  0.0072   27.6   0.0   10  391-400    33-42  (224)\n",
      "  7 QGI59602.1|GH16_22|78-291        5.3       5  0.0073   27.5   0.0   18  581-598   204-221 (224)\n",
      "  8 AQA16748.1|GH5_51.hmm|7.4e-189   4.9     5.5  0.0081   28.6   0.0   37  644-680   253-291 (351)\n",
      "  9 CCF60459.1|GH5_12.hmm|1.2e-238   3.3     9.1   0.013   28.3   0.0   27  357-383   298-324 (541)\n",
      " 10 ACI55886.1|GH25|58-236|2.7e-60   3.0      10   0.015   22.2   0.0   41  594-634    18-61  (174)\n",
      "\n",
      "No 1\n",
      ">AAA91086.1|GH48|150-238|4.7e-10\n",
      "Probab=100.00  E-value=6.7e-195  Score=1475.15  Aligned_cols=608  Identities=60%  Similarity=1.105  Sum_probs=604.0  Template_Neff=2.700\n",
      "\n",
      "Q tr|G8M3C3|G8M3   31 FKDRFNYMYNKIHDPANGYFDSEGIPYHSVETLCVEAPDYGHESTSEAASYYAWLEAVNGKLNGKWSGLTEAWNVVEKYF  110 (728)\n",
      "Q Consensus        31 fkdrfnymynkihdpangyfdsegipyhsvetlcveapdyghestseaasyyawleavngklngkwsglteawnvvekyf  110 (728)\n",
      "                      |.||||+||+|||||+|||||++||||||||||||||||||||||||||||++|||||||+|||||++|++||++||+||\n",
      "T Consensus         1 Y~~rFl~lY~kI~dp~nGYFS~~GiPYHsvETlivEAPDyGHeTTSEA~SY~~WLeAmyg~itgd~s~~~~AW~~mE~y~   80 (620)\n",
      "T AAA91086.1|GH4    1 YKQRFLELYNKIHDPANGYFSPEGIPYHSVETLIVEAPDYGHETTSEAYSYYVWLEAMYGKITGDWSGFNKAWDTMEKYI   80 (620)\n",
      "Confidence            78999999999999999999999999999999999999999999999999999999999999999999999999999999\n",
      "\n",
      "\n",
      "Q tr|G8M3C3|G8M3  111 IPSESIQKGMNRYNPSSPAGYADEFPLPDDYPAQIQSNVTVGQDPIHQELVSAYNTYAMYGMHWLVDVDNWYGYGT----  186 (728)\n",
      "Q Consensus       111 ipsesiqkgmnrynpsspagyadefplpddypaqiqsnvtvgqdpihqelvsayntyamygmhwlvdvdnwygygt----  186 (728)\n",
      "                      ||++++||+|+.|||++|||||||+++|++||++|+++++||+|||++||+++||+++||+||||||||||||||+    \n",
      "T Consensus        81 IP~~~~Qp~~~~Ynp~~pAtya~E~~~P~~YPs~l~~~~~vG~DPi~~eL~saYgt~~iY~MHWLlDVDN~YGfG~~g~~  160 (620)\n",
      "T AAA91086.1|GH4   81 IPSHQDQPTMSSYNPSSPATYAPEYDTPSQYPSQLDFNVPVGQDPIANELKSAYGTDDIYGMHWLLDVDNWYGFGNLGDG  160 (620)\n",
      "Confidence            9999999999999999999999999999999999999999999999999999999999999999999999999999    \n",
      "\n",
      "\n",
      "Q tr|G8M3C3|G8M3  187 GTNCTFINTYQRGEQESVFETVPHPSIEEFKYGGRQGFSDLFTAG-ETQPKWAFTIASDADGRLIQVQYWANKWAKEQGQ  265 (728)\n",
      "Q Consensus       187 gtnctfintyqrgeqesvfetvphpsieefkyggrqgfsdlftag-etqpkwaftiasdadgrliqvqywankwakeqgq  265 (728)\n",
      "                      +++|+||||||||+||||||||||||||+|||||+||||+||++| ++++|||||||||||||||||+|||++||+|||+\n",
      "T Consensus       161 ~~~psyINTfQRG~qESvWeTvp~P~~d~fk~Gg~nGfldlFt~d~~ya~QwkYTnApDADARavQa~YwA~~Wa~e~G~  240 (620)\n",
      "T AAA91086.1|GH4  161 TSGPSYINTFQRGEQESVWETVPHPSCEEFKYGGPNGFLDLFTKDSSYAKQWRYTNAPDADARAVQAAYWANQWAKEQGK  240 (620)\n",
      "Confidence            899999999999999999999999999999999999999999999 9999999999999999999999999999999999\n",
      "\n",
      "\n",
      "Q tr|G8M3C3|G8M3  266 --NLSTLNAKAAKLGDYLRYSMFDKYFMKIG--AQGKTPASGYDSCHYLLAWYYAWGGAIAG-DWSWKIGCSHVHWGYQA  340 (728)\n",
      "Q Consensus       266 --nlstlnakaaklgdylrysmfdkyfmkig--aqgktpasgydschyllawyyawggaiag-dwswkigcshvhwgyqa  340 (728)\n",
      "                        +|+++++||+||||||||+||||||||||  +.+|++|+||||||||||||++|||++++ +|+|||||||+||||||\n",
      "T Consensus       241 ~~~is~~~~KAaKmGDyLRY~mfDKYfkkiG~~~~s~~ag~GkdSaHYLlsWY~aWGG~~~~~~WaWrIG~Sh~H~GYQN  320 (620)\n",
      "T AAA91086.1|GH4  241 ESEISSTVAKAAKMGDYLRYAMFDKYFKKIGVGPSSCPAGTGKDSAHYLLSWYYAWGGALDGSGWAWRIGSSHAHFGYQN  320 (620)\n",
      "Confidence              99999999999999999999999999999  89999999999999999999999999999 99999999999999999\n",
      "\n",
      "\n",
      "Q tr|G8M3C3|G8M3  341 PLAAYALANDPDLKPKSANGAKDWNSSFKRQVELYAWLQSAEGAIAGGVTNSVGGQYKSY-NGASTFYDMAYTYAPVYAD  419 (728)\n",
      "Q Consensus       341 plaayalandpdlkpksangakdwnssfkrqvelyawlqsaegaiaggvtnsvggqyksy-ngastfydmaytyapvyad  419 (728)\n",
      "                      ||||||||++++|||||+|+++||++||+||||||+||||+||+|||||||||+|+|++| +|++|||||+|+++|||||\n",
      "T Consensus       321 P~AAyaLs~~~~lkPks~ta~~DW~~SL~RQlEfy~wLQS~eG~iAGGaTNSW~G~Y~~~Psg~~TFygM~Yd~~PVY~D  400 (620)\n",
      "T AAA91086.1|GH4  321 PLAAYALSNDSDLKPKSPTAASDWAKSLDRQLEFYQWLQSAEGAIAGGATNSWNGRYETYPSGTSTFYGMAYDEHPVYHD  400 (620)\n",
      "Confidence            999999999999999999999999999999999999999999999999999999999999 9999999999999999999\n",
      "\n",
      "\n",
      "Q tr|G8M3C3|G8M3  420 PPSNNWFGMQAWSMQRMCEVYYETGDSLAKEICDKWVAWAESVCEADIEAGTWKIPATLEWSGQPDTWRGTKPSNNNLHC  499 (728)\n",
      "Q Consensus       420 ppsnnwfgmqawsmqrmcevyyetgdslakeicdkwvawaesvceadieagtwkipatlewsgqpdtwrgtkpsnnnlhc  499 (728)\n",
      "                      ||||+|||||+|+|||||||||+|||++||+||||||+|++++|+|+ ++|+|+||++|+|+|||||||+++++|+||||\n",
      "T Consensus       401 PpSN~WfG~Q~Wsm~RvAeyYY~tGD~~ak~ildKWv~W~~~~~~~~-~dg~~~iPs~L~WsGqPDtW~gs~~~N~~lhv  479 (620)\n",
      "T AAA91086.1|GH4  401 PPSNRWFGMQAWSMQRVAEYYYVTGDARAKAILDKWVAWVKSNTTVN-SDGTFQIPSTLEWSGQPDTWNGSYTGNPNLHV  479 (620)\n",
      "Confidence            99999999999999999999999999999999999999999999999 88999999999999999999999999999999\n",
      "\n",
      "\n",
      "Q tr|G8M3C3|G8M3  500 KVVNYGNDIGITGSLANAFLFYDQATQRWNGNTTLGKKAADKALAMLQVVWDTCRDQYGVGVKETNESLNRIFTQEVFIP  579 (728)\n",
      "Q Consensus       500 kvvnygndigitgslanaflfydqatqrwngnttlgkkaadkalamlqvvwdtcrdqygvgvketneslnriftqevfip  579 (728)\n",
      "                      +|+++|+|||||+||||||+||||++|+++ |    ++||++||+|||+||++|||++||+++|+|+||+||++++||||\n",
      "T Consensus       480 ~V~~yg~dvGva~s~A~tL~yYAa~sg~~~-d----~~ak~~Ak~LLD~~w~~~~d~~Gvs~~E~r~dy~Rf~d~~VYiP  554 (620)\n",
      "T AAA91086.1|GH4  480 TVTDYGQDVGVAASLAKTLMYYAAASGKYG-D----TAAKNLAKQLLDAMWKNYRDDKGVSTPETRGDYKRFFDQEVYIP  554 (620)\n",
      "Confidence            999999999999999999999999999888 7    88999999999999999999999999999999999999999999\n",
      "\n",
      "\n",
      "Q tr|G8M3C3|G8M3  580 AGWTGKMPNGDVIQQGVKFIDIRSKYKDDPWYEGLKKQAEQGIPFEYTLHRFWHQVDYAVALGIA  644 (728)\n",
      "Q Consensus       580 agwtgkmpngdviqqgvkfidirskykddpwyeglkkqaeqgipfeytlhrfwhqvdyavalgia  644 (728)\n",
      "                      +||+|+|||||+|++|+|||||||+||+||+|++||++|++|++|+|+|||||+|+|||||+|+.\n",
      "T Consensus       555 ~gwtG~mPnGD~I~~g~tFl~IRs~Yk~Dp~w~kvq~~l~gG~~p~f~YHRFWaQ~diA~A~g~y  619 (620)\n",
      "T AAA91086.1|GH4  555 SGWTGTMPNGDVIKSGATFLDIRSKYKQDPDWPKVEAYLNGGAAPEFTYHRFWAQADIAMANGTY  619 (620)\n",
      "Confidence            99999999999999999999999999999999999999999999999999999999999999863\n",
      "\n",
      "\n",
      "No 2\n",
      ">lcl|consensus\n",
      "Probab=91.79  E-value=0.00051  Score=37.45  Aligned_cols=20  Identities=55%  Similarity=0.811  Sum_probs=11.9  Template_Neff=4.300\n",
      "\n",
      "Q tr|G8M3C3|G8M3  668 DINFDGDINSIDYALLKAHL  687 (728)\n",
      "Q Consensus       668 dinfdgdinsidyallkahl  687 (728)\n",
      "                      |+|-||.+|++|++++|.++\n",
      "T Consensus         1 DvN~DG~Vna~D~~~l~~~l   20 (21)\n",
      "T lcl|consensus_    1 DVNGDGKVNALDLALLKKYL   20 (21)\n",
      "Confidence            45566666666666665554\n",
      "\n",
      "\n",
      "No 3\n",
      ">ABN51673.1|GH124|2-334|2.6e-219\n",
      "Probab=52.47  E-value=0.096  Score=40.07  Aligned_cols=66  Identities=35%  Similarity=0.533  Sum_probs=48.5  Template_Neff=1.400\n",
      "\n",
      "Q tr|G8M3C3|G8M3  663 DIKLGDINFDGDINSIDYALLKAHLLGINKLSGDAL-KAADVDQNGDVNSIDYAKMKSYLLGISKDF  728 (728)\n",
      "Q Consensus       663 diklgdinfdgdinsidyallkahllginklsgdal-kaadvdqngdvnsidyakmksyllgiskdf  728 (728)\n",
      "                      .+..||.|-||-+|--||.|+|..|.-|.+...+.- -..+++....++.+|-.-+|.|||.+-++|\n",
      "T Consensus        19 kav~GD~n~dgvv~isd~vl~k~~l~~~a~~~a~~d~w~g~vN~dd~I~D~d~~~~kryll~mir~~   85 (318)\n",
      "T ABN51673.1|GH1   19 KAVIGDVNADGVVNISDYVLMKRILRIIADFPADDDMWVGDVNGDDVINDIDCNYLKRYLLHMIREF   85 (318)\n",
      "Confidence            567899999999999999999997766666543321 123444445577888888999999876553\n",
      "\n",
      "\n",
      "No 4\n",
      ">CAR68154.1|GH88|62-388|4.9e-137\n",
      "Probab=10.51  E-value=2  Score=30.74  Aligned_cols=43  Identities=19%  Similarity=0.234  Sum_probs=34.3  Template_Neff=5.400\n",
      "\n",
      "Q tr|G8M3C3|G8M3  421 PSNNWFGMQAWSMQRMCEVYYETGDSLAKEICDKWVAWAESVC  463 (728)\n",
      "Q Consensus       421 psnnwfgmqawsmqrmcevyyetgdslakeicdkwvawaesvc  463 (728)\n",
      "                      .+..|=-=|+|.|==.|..|..|||++--++-.+=+.+++++.\n",
      "T Consensus       181 d~S~WsRGQAWaiYG~a~~yr~t~d~~yL~~A~~~a~yfl~~l  223 (329)\n",
      "T CAR68154.1|GH8  181 DDSAWARGQAWAIYGFALAYRYTKDPEYLDTAKKVADYFLNRL  223 (329)\n",
      "Confidence            3678999999999999999999999995555555555555555\n",
      "\n",
      "\n",
      "No 5\n",
      ">ACY49347.1|GH105|46-385|1.1e-131\n",
      "Probab=6.37  E-value=4  Score=28.22  Aligned_cols=60  Identities=25%  Similarity=0.330  Sum_probs=50.1  Template_Neff=6.200\n",
      "\n",
      "Q tr|G8M3C3|G8M3  324 DWSWKIGCSHVHWGYQAPLAAYALANDPDLKPKSANGAKDWNSSFKRQVELYAWLQSAEG  383 (728)\n",
      "Q Consensus       324 dwswkigcshvhwgyqaplaayalandpdlkpksangakdwnssfkrqvelyawlqsaeg  383 (728)\n",
      "                      .|+=.-|.|..+.|==|=-.+.||...-++-|+.........+.|++|++=..=+|+.+|\n",
      "T Consensus       169 ~wa~~t~~s~~fW~RgnGW~~~aL~~~L~~lP~~~p~r~~l~~~~~~~~~al~~~Qd~~G  228 (329)\n",
      "T ACY49347.1|GH1  169 NWADPTGGSPAFWGRGNGWVAMALVDVLELLPEDHPDRRFLIDILKEQAAALAKYQDESG  228 (329)\n",
      "Confidence            444445667777777788889999999999998888889999999999998888999776\n",
      "\n",
      "\n",
      "No 6\n",
      ">QGI59602.1|GH16_22|78-291\n",
      "Probab=5.38  E-value=4.9  Score=27.58  Aligned_cols=10  Identities=30%  Similarity=0.245  Sum_probs=6.0  Template_Neff=3.000\n",
      "\n",
      "Q tr|G8M3C3|G8M3  391 NSVGGQYKSY  400 (728)\n",
      "Q Consensus       391 nsvggqyksy  400 (728)\n",
      "                      ||-+..|-..\n",
      "T Consensus        33 NS~nNvyie~   42 (224)\n",
      "T QGI59602.1|GH1   33 NSPNNVYIEK   42 (224)\n",
      "Confidence            6666666555\n",
      "\n",
      "\n",
      "No 7\n",
      ">QGI59602.1|GH16_22|78-291\n",
      "Probab=5.33  E-value=5  Score=27.55  Aligned_cols=18  Identities=28%  Similarity=0.730  Sum_probs=9.8  Template_Neff=3.000\n",
      "\n",
      "Q tr|G8M3C3|G8M3  581 GWTGKMPNGDVIQQGVKF  598 (728)\n",
      "Q Consensus       581 gwtgkmpngdviqqgvkf  598 (728)\n",
      "                      .|+|.|.-|+...-++..\n",
      "T Consensus       204 ~WsGnM~vg~sa~lqIqW  221 (224)\n",
      "T QGI59602.1|GH1  204 SWSGNMSVGDSAYLQIQW  221 (224)\n",
      "Confidence            366666666654444333\n",
      "\n",
      "\n",
      "No 8\n",
      ">AQA16748.1|GH5_51.hmm|7.4e-189|58-409\n",
      "Probab=4.89  E-value=5.5  Score=28.57  Aligned_cols=37  Identities=32%  Similarity=0.524  Sum_probs=28.0  Template_Neff=2.200\n",
      "\n",
      "Q tr|G8M3C3|G8M3  644 AEIFGYKPPK--GGSGGGETGDIKLGDINFDGDINSIDY  680 (728)\n",
      "Q Consensus       644 aeifgykppk--ggsgggetgdiklgdinfdgdinsidy  680 (728)\n",
      "                      |..+||.-|+  |.+|-|||.|.+..|+.-..-.+.++-\n",
      "T Consensus       253 aHfYgYTGP~htGatg~get~dpRY~Dl~~~~l~~~l~~  291 (351)\n",
      "T AQA16748.1|GH5  253 AHFYGYTGPNHTGATGIGETHDPRYRDLSPAELAAVLDD  291 (351)\n",
      "Confidence            5678998775  677789999999999877655555443\n",
      "\n",
      "\n",
      "No 9\n",
      ">CCF60459.1|GH5_12.hmm|1.2e-238|14-567\n",
      "Probab=3.29  E-value=9.1  Score=28.28  Aligned_cols=27  Identities=26%  Similarity=0.540  Sum_probs=17.1  Template_Neff=4.100\n",
      "\n",
      "Q tr|G8M3C3|G8M3  357 SANGAKDWNSSFKRQVELYAWLQSAEG  383 (728)\n",
      "Q Consensus       357 sangakdwnssfkrqvelyawlqsaeg  383 (728)\n",
      "                      .|.+++-|.+.-+|+=.-|-|-.+++-\n",
      "T Consensus       298 np~G~saWl~~~~~~d~~ygw~r~~~w  324 (541)\n",
      "T CCF60459.1|GH5  298 NPKGVSAWLSGEERDDKKYGWKRDPEW  324 (541)\n",
      "Confidence            345667777777777666777655443\n",
      "\n",
      "\n",
      "No 10\n",
      ">ACI55886.1|GH25|58-236|2.7e-60\n",
      "Probab=3.03  E-value=10  Score=22.20  Aligned_cols=41  Identities=20%  Similarity=0.114  Sum_probs=30.1  Template_Neff=7.700\n",
      "\n",
      "Q tr|G8M3C3|G8M3  594 QGVKFIDIRSKY---KDDPWYEGLKKQAEQGIPFEYTLHRFWHQ  634 (728)\n",
      "Q Consensus       594 qgvkfidirsky---kddpwyeglkkqaeqgipfeytlhrfwhq  634 (728)\n",
      "                      +|..|.-||.-+   -.||.|..=-+....-..|.=.||-+.+.\n",
      "T Consensus        18 ~gi~Fv~ikateG~~~~D~~f~~n~~~a~~aGl~~G~Yhf~~~~   61 (174)\n",
      "T ACI55886.1|GH2   18 SGVDFVIIKATEGTSYVDPYFASNWAGARAAGLPVGAYHFARPC   61 (174)\n",
      "Confidence            389999999765   36888876555555556788889988854\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "- 12:48:16.066 INFO: Premerge done\n",
      "\n",
      "- 12:48:16.066 INFO: Realigning 10 HMM-HMM alignments using Maximum Accuracy algorithm\n",
      "\n",
      "- 12:48:16.115 INFO: 4 sequences belonging to 4 database HMMs found with an E-value < 0.001\n",
      "\n",
      "- 12:48:16.115 INFO: Number of effective sequences of resulting query HMM: Neff = 2.41642\n",
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "!wget -O protein3.fasta https://www.uniprot.org/uniprot/G8M3C3.fasta\n",
    "\n",
    "dataset_path = 'protein3.fasta'\n",
    "sequence_utils.hhblits(dataset_path,database='dbCAN-fam-V9', data_dir=data_dir)\n",
    "\n",
    "#open the results and print them\n",
    "f = open(\"protein3.hhr\", \"r\")\n",
    "print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zi3Bkj1-TfB_"
   },
   "source": [
    "We can see that the exact protein was found in dbCAN in hit 1, but also some highly related proteins were found in hits 1-5. This query.a3m MSA can then be useful if we want to search a larger database like UniProt or Uniclust because it includes this more diverse selection of related protein sequences. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uztsqiPLvhVB"
   },
   "source": [
    "## Other hh-suite functions\n",
    "hhsuite contains other functions which may be useful if you are working with MSA or HMMs. For more detailed information, see the documentation at https://github.com/soedinglab/hh-suite/wiki\n",
    "\n",
    "hhmake: Build an HMM from an input MSA\n",
    "\n",
    "hhfilter: Filter an MSA by max sequence identity, coverage, and other criteria\n",
    "\n",
    "hhalign: Calculate pairwise alignments etc. for two HMMs/MSAs\n",
    "\n",
    "hhconsensus: Calculate the consensus sequence for an A3M/FASTA input file\n",
    "\n",
    "reformat.pl: Reformat one or many MSAs\n",
    "\n",
    "addss.pl: Add PSIPRED predicted secondary structure to an MSA or HHM file\n",
    "\n",
    "hhmakemodel.pl: Generate MSAs or coarse 3D models from HHsearch or HHblits \n",
    "results\n",
    "\n",
    "hhmakemodel.py: Generates coarse 3D models from HHsearch or HHblits results and modifies cif files such that they are compatible with MODELLER\n",
    "\n",
    "hhsuitedb.py: Build HHsuite database with prefiltering, packed MSA/HMM, and index files\n",
    "\n",
    "splitfasta.pl: Split a multiple-sequence FASTA file into multiple single-sequence files\n",
    "\n",
    "renumberpdb.pl: Generate PDB file with indices renumbered to match input sequence indices\n",
    "\n",
    "HHPaths.pm: Configuration file with paths to the PDB, BLAST, PSIPRED etc.\n",
    "mergeali.pl: Merge MSAs in A3M format according to an MSA of their seed sequences\n",
    "\n",
    "pdb2fasta.pl: Generate FASTA sequence file from SEQRES records of globbed pdb files\n",
    "\n",
    "cif2fasta.py: Generate a FASTA sequence from the pdbx_seq_one_letter_code entry of the entity_poly of globbed cif files\n",
    "\n",
    "pdbfilter.pl: Generate representative set of PDB/SCOP sequences from pdb2fasta.pl output\n",
    "\n",
    "pdbfilter.py: Generate representative set of PDB/SCOP sequences from cif2fasta.py output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "References:\n",
    "\n",
    "[1] Steinegger M, Meier M, Mirdita M, Vöhringer H, Haunsberger S J, and Söding J (2019) HH-suite3 for fast remote homology detection and deep protein annotation, BMC Bioinformatics, 473. doi: 10.1186/s12859-019-3019-7\n",
    "\n",
    "[2] Kunzmann, P., Mayer, B.E. & Hamacher, K. Substitution matrix based color schemes for sequence alignment visualization. BMC Bioinformatics 21, 209 (2020). https://doi.org/10.1186/s12859-020-3526-6\n",
    "\n",
    "[3]Identifying DNA and protein patterns with statistically significant alignments of multiple \n",
    "sequences. https://www.researchgate.net/publication/12812078_\n",
    "\n",
    "[4]https://github.com/soedinglab/hh-suite/wiki#what-are-hmm-hmm-comparisons-and-why-are-they-so-powerful"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Congratulations! Time to join the Community!\n",
    "\n",
    "Congratulations on completing this tutorial notebook! If you enjoyed working through the tutorial, and want to continue working with DeepChem, we encourage you to finish the rest of the tutorials in this series. You can also help the DeepChem community in the following ways:\n",
    "\n",
    "## Star DeepChem on [GitHub](https://github.com/deepchem/deepchem)\n",
    "This helps build awareness of the DeepChem project and the tools for open source drug discovery that we're trying to build.\n",
    "\n",
    "## Join the DeepChem Gitter\n",
    "The DeepChem [Gitter](https://gitter.im/deepchem/Lobby) hosts a number of scientists, developers, and enthusiasts interested in deep learning for the life sciences. Join the conversation!"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "hh-suite Tutorial.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
